diff --git a/.gitbook/assets/9 Dimensions (1).jpg b/.gitbook/assets/9 Dimensions (1).jpg new file mode 100644 index 00000000..e3b5b209 Binary files /dev/null and b/.gitbook/assets/9 Dimensions (1).jpg differ diff --git a/.gitbook/assets/9 Dimensions.jpg b/.gitbook/assets/9 Dimensions.jpg new file mode 100644 index 00000000..dbdf7cc8 Binary files /dev/null and b/.gitbook/assets/9 Dimensions.jpg differ diff --git a/.gitbook/assets/9 Dimensions.png b/.gitbook/assets/9 Dimensions.png new file mode 100644 index 00000000..7baee6ee Binary files /dev/null and b/.gitbook/assets/9 Dimensions.png differ diff --git a/.gitbook/assets/9-dimensions (2).jpg b/.gitbook/assets/9-dimensions (2).jpg new file mode 100644 index 00000000..e3b5b209 Binary files /dev/null and b/.gitbook/assets/9-dimensions (2).jpg differ diff --git a/.gitbook/assets/APIs.gif b/.gitbook/assets/APIs.gif new file mode 100644 index 00000000..c9629778 Binary files /dev/null and b/.gitbook/assets/APIs.gif differ diff --git a/.gitbook/assets/Auto Rules.png b/.gitbook/assets/Auto Rules.png new file mode 100644 index 00000000..53e78baa Binary files /dev/null and b/.gitbook/assets/Auto Rules.png differ diff --git a/.gitbook/assets/Behavior.jpg b/.gitbook/assets/Behavior.jpg new file mode 100644 index 00000000..1a830f0a Binary files /dev/null and b/.gitbook/assets/Behavior.jpg differ diff --git a/.gitbook/assets/DQ-Admin-Console-1.png b/.gitbook/assets/DQ-Admin-Console-1.png new file mode 100644 index 00000000..58d41ab7 Binary files /dev/null and b/.gitbook/assets/DQ-Admin-Console-1.png differ diff --git a/.gitbook/assets/DQ-Admin-Console-2.png.png b/.gitbook/assets/DQ-Admin-Console-2.png.png new file mode 100644 index 00000000..5685f737 Binary files /dev/null and b/.gitbook/assets/DQ-Admin-Console-2.png.png differ diff --git a/.gitbook/assets/DQ-Admin-Console-3.png b/.gitbook/assets/DQ-Admin-Console-3.png new file mode 100644 index 00000000..df35ca6b Binary files /dev/null and b/.gitbook/assets/DQ-Admin-Console-3.png differ diff --git a/.gitbook/assets/DQ-Admin-Console-4.png b/.gitbook/assets/DQ-Admin-Console-4.png new file mode 100644 index 00000000..ff348fc2 Binary files /dev/null and b/.gitbook/assets/DQ-Admin-Console-4.png differ diff --git a/.gitbook/assets/DQ-Admin-Console-5 (1).png b/.gitbook/assets/DQ-Admin-Console-5 (1).png new file mode 100644 index 00000000..f3486fcc Binary files /dev/null and b/.gitbook/assets/DQ-Admin-Console-5 (1).png differ diff --git a/.gitbook/assets/DQ-Admin-Console-5.png b/.gitbook/assets/DQ-Admin-Console-5.png new file mode 100644 index 00000000..e9124c8f Binary files /dev/null and b/.gitbook/assets/DQ-Admin-Console-5.png differ diff --git a/.gitbook/assets/DQ-Agent-Config-Hint-1.png b/.gitbook/assets/DQ-Agent-Config-Hint-1.png new file mode 100644 index 00000000..450c2ad6 Binary files /dev/null and b/.gitbook/assets/DQ-Agent-Config-Hint-1.png differ diff --git a/.gitbook/assets/DQ-Connection-1.png b/.gitbook/assets/DQ-Connection-1.png new file mode 100644 index 00000000..afd18ce6 Binary files /dev/null and b/.gitbook/assets/DQ-Connection-1.png differ diff --git a/.gitbook/assets/DQ-Connection-2.png b/.gitbook/assets/DQ-Connection-2.png new file mode 100644 index 00000000..da388605 Binary files /dev/null and b/.gitbook/assets/DQ-Connection-2.png differ diff --git a/.gitbook/assets/DQ-Connection-3.png b/.gitbook/assets/DQ-Connection-3.png new file mode 100644 index 00000000..9666405c Binary files /dev/null and b/.gitbook/assets/DQ-Connection-3.png differ diff --git a/.gitbook/assets/DQ-Connection-4.png b/.gitbook/assets/DQ-Connection-4.png new file mode 100644 index 00000000..d6d2d5e7 Binary files /dev/null and b/.gitbook/assets/DQ-Connection-4.png differ diff --git a/.gitbook/assets/DQ-Login.png b/.gitbook/assets/DQ-Login.png new file mode 100644 index 00000000..870ef35e Binary files /dev/null and b/.gitbook/assets/DQ-Login.png differ diff --git a/.gitbook/assets/Data accuracy with precision.jpg b/.gitbook/assets/Data accuracy with precision.jpg new file mode 100644 index 00000000..29fd6e46 Binary files /dev/null and b/.gitbook/assets/Data accuracy with precision.jpg differ diff --git a/.gitbook/assets/Dupes.jpg b/.gitbook/assets/Dupes.jpg new file mode 100644 index 00000000..7dc3fa56 Binary files /dev/null and b/.gitbook/assets/Dupes.jpg differ diff --git a/.gitbook/assets/Insurance data.jpg b/.gitbook/assets/Insurance data.jpg new file mode 100644 index 00000000..6492817c Binary files /dev/null and b/.gitbook/assets/Insurance data.jpg differ diff --git a/.gitbook/assets/List View with impact.jpg b/.gitbook/assets/List View with impact.jpg new file mode 100644 index 00000000..0708fcd4 Binary files /dev/null and b/.gitbook/assets/List View with impact.jpg differ diff --git a/.gitbook/assets/List View.jpg b/.gitbook/assets/List View.jpg new file mode 100644 index 00000000..aa1a09be Binary files /dev/null and b/.gitbook/assets/List View.jpg differ diff --git a/.gitbook/assets/Outliers.jpg b/.gitbook/assets/Outliers.jpg new file mode 100644 index 00000000..a4be6498 Binary files /dev/null and b/.gitbook/assets/Outliers.jpg differ diff --git a/.gitbook/assets/Owl Dataset (2).csv b/.gitbook/assets/Owl Dataset (2).csv new file mode 100644 index 00000000..e37e95a4 --- /dev/null +++ b/.gitbook/assets/Owl Dataset (2).csv @@ -0,0 +1,9 @@ +"Abcacc_now_delinq","#acc_open_past_24mths","Abcaddr_state","#all_util","#annual_inc","Abcannual_inc_joint","#application_type","#avg_cur_bal","#bc_open_to_buy","#.#bc_util","#.#chargeoff_within_12_mths","#.#collection_recovery_fee","#.#collections_12_mths_ex_med","Abccredit_rating","Abcdebt_settlement_flag","Abcdebt_settlement_flag_date","Abcdeferral_term","#delinq_2yrs","#delinq_amnt","Abcdesc","Abcdisbursement_method","#.#dti","#.#dti_joint","Abcearliest_cr_line","Abcemp_length","Abcemp_title","#funded_amnt","#funded_amnt_inv","Abcgrade","Abchardship_amount","Abchardship_dpd","Abchardship_end_date","Abchardship_flag","Abchardship_last_payment_amount","Abchardship_length","Abchardship_loan_status","Abchardship_payoff_balance_amount","#hardship_reason","Abchardship_start_date","Abchardship_status","Abchardship_type","Abchome_ownership","#il_util","#initial_list_status","#inq_fi","#inq_last_12m","#inq_last_6mths","#.#installment","#.#int_rate","Abcissue_d","Abclast_credit_pull_d","#last_pymnt_amnt","last_pymnt_d","#loan_amnt","Abcloan_status","#max_bal_bc","#member_id","#mort_acc","#.#mo_sin_old_il_acct","#mo_sin_old_rev_tl_op","#mo_sin_rcnt_rev_tl_op","#mo_sin_rcnt_tl","#mths_since_last_delinq","Abcmths_since_last_major_derog","#mths_since_last_record","#mths_since_rcnt_il","#mths_since_recent_bc","#mths_since_recent_bc_dlq","#mths_since_recent_inq","#mths_since_recent_revol_delinq","#next_pymnt_d","#num_accts_ever_120_pd","#num_actv_bc_tl","#num_actv_rev_tl","#num_bc_sats","#num_bc_tl","#num_il_tl","#num_op_rev_tl","#num_rev_accts","#num_rev_tl_bal_gt_0","#num_sats","#num_tl_120dpd_2m","#num_tl_30dpd","#num_tl_90g_dpd_24m","#num_tl_op_past_12m","#open_acc_6m","#open_act_il","#open_il_12m","#open_il_24m","#open_rv_12m","#open_rv_24m","Abcorig_projected_additional_accrued_interest","#.#out_prncp","#.#out_prncp_inv","Abcpayment_plan_start_date","#.#pct_tl_nvr_dlq","#.#percent_bc_gt_75","Abcpolicy_code","#.#pub_rec_bankruptcies","Abcpurpose","Abcpymnt_plan","#.#rate","#.#recoveries","#revol_bal_joint","#.#revol_util","#.#sec_app_chargeoff_within_12_mths","#sec_app_collections_12_mths_ex_med","#sec_app_earliest_cr_line","#sec_app_inq_last_6mths","#sec_app_mort_acc","#sec_app_mths_since_last_major_derog","#sec_app_num_rev_accts","Abcsec_app_open_acc","#.#sec_app_open_act_il","#.#sec_app_revol_util","Abcsettlement_amount","Abcsettlement_date","Abcsettlement_percentage","Abcsettlement_status","Abcsettlement_term","Abcssn_num","Abcsub_grade","#.#tax_liens","Abcterm","Abctitle","#total_acc","#total_bal_ex_mort","#total_bal_il","#total_bc_limit","#total_cu_tl","#total_il_high_credit_limit","Abctotal_pymnt","#.#total_pymnt_inv","#.#total_rec_int","#.#total_rec_late_fee","#.#total_rec_prncp","#total_rev_hi_lim","#.#tot_coll_amt","Abctot_cur_bal","#.#tot_hi_cred_lim","Abcurl","Abcverification_status","Abcverification_status_joint","Abczip_code" +"105000","2","TX","2","30000","69","0","0","9","6.0","5204.0","748.78000","890.95000","POOR","","","","0","6948","","","67.68000","1.0","Jun-95","7 years","overnight supervisor","24000","24000","D","","","","0","","","","","25","","N","2","RENT","1","14906","3780","25200","2","890.95000","19.92000","18-Dec","1/1/19","0","2019-01-01 00:00:00.0","24000","Current","80","83","225","60.70000","0","0","42","61","19-Mar","","2","7","6","0","7","0","","2","","1","7","11","7","7","3","11","11","11","14","0","0","0","67646","1","6","52740","","59.20000","15.0","","0.0","0.0","2/1/19","3.0","credit_card","n","8.50000","993.28000","67646","14.0","73.40000","2","17700","66148","42959","25","24","Apr-90","0.0","0.0","","N","","DirectPay","","","D3","93.30000","36 months","Credit card refinancing","0","0","1","0","74","91348","w","23006.72000","1742.06000","1742.06000","23006.72000","5","35.52000","","28.60000","","Verified","Joint App","752xx" +"","1","VA","2","70200","7","2","0","4","4.0","21681.0","225.0","302.67000","FAIR","","","","4","3131","","","3.62000","1.0","5-Dec","7 years","DIRECT CARE PROVIDER","8500","8500","C","","","","","","","","","","","N","","MORTGAGE","0","5721","4233","21124","1","302.67000","16.91000","18-Dec","1/1/19","0","2019-01-01 00:00:00.0","8500","Current","0","100","156","46.0","0","0","146","6","19-Mar","","0","9","9","1","12","0","27","1","6","0","4","8","4","6","9","8","23","2","9","","0","3632","195134","0","47","0","","27.0","33.0","","0.0","2.0","12/1/18","2.0","debt_consolidation","n","6.50000","368.36000","5721","9.0","","","12524","0","","","","","","","","N","","Cash","","","C5","88.0","36 months","Debt consolidation","0","0","0","0","27","221124","w","8131.64000","593.36000","593.36000","8131.64000","4","","","0.0","","Not Verified","Individual","226xx" +"","0","NY","1","60000","","0","3","0","5.0","28962.0","70.51000","104.03000","FAIR","","","","0","5402","","","21.36000","1.0","6-Oct","4 years","Banker","3000","3000","C","","","","","","","","","","","N","","MORTGAGE","3","11026","3407","22200","0","104.03000","15.02000","18-Dec","1/1/19","0","2019-01-01 00:00:00.0","3000","Current","97","78","140","53.0","0","0","146","","19-Mar","","4","9","2","2","28","0","","","","0","4","7","4","6","7","8","11","7","13","0","0","0","376508","1","2","49048","","49.70000","20.0","","0.0","0.0","2/1/19","3.0","home_improvement","n","6.50000","133.79000","60074","13.0","","","11500","50524","","","","","","","","N","","Cash","","","C3","95.0","36 months","Home improvement","0","0","2","0","83","400137","w","2866.21000","204.30000","204.30000","2866.21000","1","","","25.0","","Source Verified","Individual","104xx" +"","2","GA","3","60000","","0","0","5","13.0","2678.0","378.93000","528.40000","FAIR","","","","0","3158","","","18.86000","1.0","8-Aug","3 years","asst.mgr","15000","15000","C","","","","","","","","","","","N","","RENT","6","5311","2155","10800","1","528.40000","16.14000","18-Dec","1/1/19","0","2019-01-01 00:00:00.0","15000","Current","78","95","118","55.50000","0","0","124","","19-Mar","","2","1","1","0","1","0","","1","","0","3","7","4","5","14","9","10","7","11","0","0","0","29454","2","7","24143","","49.20000","24.0","","0.0","0.0","2/1/19","7.0","credit_card","n","6.50000","657.69000","29454","11.0","","","7100","31084","","","","","","","","N","","Cash","","","C4","100.0","36 months","Credit card refinancing","0","0","4","0","70","41884","w","14342.31000","1036.62000","1036.62000","14342.31000","7","","","0.0","","Source Verified","Individual","301xx" +"null","1","PA","4","20000","null","0","0","2","9.0","224.0","73.82000","132.48000","GOOD","null","null","null","0","13204","null","null","12.24000","1.0","12-Sep","n/a","null","4000","4000","B","null","null","null","null","null","null","null","null","null","null","N","null","RENT","1","1045","646","26900","0","132.48000","11.80000","18-Dec","1/1/19","0","2019-01-01 00:00:00.0","4000","Current","80","44","56","7.0","0","0","75","null","19-Mar","null","1","4","4","0","8","0","null","10","null","0","3","4","7","10","2","18","26","4","19","0","0","500","4255","1","10","3210","null","3.90000","28.0","null","0.0","0.0","2/1/19","5.0","debt_consolidation","n","5.50000","187.21000","4255","19.0","null","null","14200","4000","null","null","null","null","null","null","null","N","null","Cash","null","null","B4","100.0","36 months","Debt consolidation","0","0","1","0","14","30900","w","3812.79000","261.03000","261.03000","3812.79000","8","null","null","0.0","null","Not Verified","Individual","151xx" +"null","2","NY","3","79365","35","0","0","4","10.0","4037.0","221.45000","397.43000","GOOD","null","null","null","0","3973","null","null","29.70000","1.0","Sep-88","10+ years","Transportation Manager","12000","12000","B","null","null","null","null","null","null","null","null","null","null","N","null","RENT","6","4957","1070","17800","2","397.43000","11.80000","18-Dec","1/1/19","0","2019-01-01 00:00:00.0","12000","Current","89","70","363","40.70000","0","0","107","25","19-Mar","null","8","1","1","0","24","0","null","1","26","1","4","11","5","5","11","12","20","11","20","0","0","0","80734","6","2","75777","null","27.80000","32.0","null","0.0","0.0","2/1/19","7.0","debt_consolidation","n","5.50000","561.61000","80734","20.0","null","null","6700","85247","null","null","null","null","null","null","null","N","null","Cash","null","null","B4","81.30000","36 months","Debt consolidation","0","0","4","0","78","103047","w","11438.39000","783.06000","783.06000","11438.39000","4","null","null","0.0","null","Source Verified","Individual","104xx" +"null","0","CO","0","82923","71","0","1","4","3.0","34150.0","134.09000","260.86000","GOOD","null","null","null","0","8676","null","null","18.55000","1.0","Jul-86","10+ years","Publications and Images Project Manager","8000","8000","B","null","null","null","null","null","null","null","null","null","null","N","null","MORTGAGE","2","16255","13624","32300","0","260.86000","10.72000","18-Dec","1/1/19","0","2019-01-01 00:00:00.0","8000","Current","95","53","141","61.10000","0","0","143","56","19-Mar","null","3","25","3","2","37","0","71","0","71","4","1","4","2","3","12","6","10","4","11","0","0","0","341504","1","15","55968","null","50.30000","25.0","null","0.0","0.0","2/1/19","1.0","car","n","5.50000","380.48000","72223","11.0","null","null","22300","59007","null","null","null","null","null","null","null","N","null","Cash","null","null","B2","80.0","36 months","Car financing","0","0","0","0","79","360944","w","7619.52000","514.57000","514.57000","7619.52000","0","null","null","50.0","null","Source Verified","Individual","802xx" +"null","0","IL","6","95000","null","0","3","5","12.0","1522.0","69.08000","265.63000","GOOD","null","null","null","0","1890","null","null","10.47000","1.0","2-Sep","2 years","Goldsmith","8000","8000","B","null","null","null","null","null","null","null","null","null","null","N","null","RENT","1","14156","3232","39400","2","260.86000","10.72000","18-Dec","1/1/19","0","2019-01-01 00:00:00.0","8000","Current","73","86","195","79.20000","0","0","135","null","19-Mar","34","2","4","4","0","8","0","null","4","null","0","3","12","3","8","6","15","28","12","17","0","0","0","25875","1","7","11719","null","35.90000","34.0","null","0.0","0.0","2/1/19","7.0","credit_card","n","5.50000","189.40000","25875","17.0","null","null","9100","16000","null","null","null","null","null","null","null","N","null","DirectPay","null","null","B2","100.0","36 months","Credit card refinancing","1","1","1","0","47","55400","w","7810.60000","258.48000","258.48000","7810.60000","11","null","null","66.70000","null","Not Verified","Individual","612xx" \ No newline at end of file diff --git a/.gitbook/assets/Owl Roles.pptx b/.gitbook/assets/Owl Roles.pptx new file mode 100644 index 00000000..714a3862 Binary files /dev/null and b/.gitbook/assets/Owl Roles.pptx differ diff --git a/.gitbook/assets/OwlDQ (1).pdf b/.gitbook/assets/OwlDQ (1).pdf new file mode 100644 index 00000000..172796f5 Binary files /dev/null and b/.gitbook/assets/OwlDQ (1).pdf differ diff --git a/.gitbook/assets/OwlDQ Healthcare.jpg b/.gitbook/assets/OwlDQ Healthcare.jpg new file mode 100644 index 00000000..1c88a8ca Binary files /dev/null and b/.gitbook/assets/OwlDQ Healthcare.jpg differ diff --git a/.gitbook/assets/OwlDQ-Framework.png b/.gitbook/assets/OwlDQ-Framework.png new file mode 100644 index 00000000..a96c711d Binary files /dev/null and b/.gitbook/assets/OwlDQ-Framework.png differ diff --git a/.gitbook/assets/Pattern.jpg b/.gitbook/assets/Pattern.jpg new file mode 100644 index 00000000..c2205373 Binary files /dev/null and b/.gitbook/assets/Pattern.jpg differ diff --git a/.gitbook/assets/Profile Management.jpg b/.gitbook/assets/Profile Management.jpg new file mode 100644 index 00000000..bd76db1b Binary files /dev/null and b/.gitbook/assets/Profile Management.jpg differ diff --git a/.gitbook/assets/Public Energy.jpg b/.gitbook/assets/Public Energy.jpg new file mode 100644 index 00000000..e169d2b4 Binary files /dev/null and b/.gitbook/assets/Public Energy.jpg differ diff --git a/.gitbook/assets/RCM Process.png b/.gitbook/assets/RCM Process.png new file mode 100644 index 00000000..e77ab693 Binary files /dev/null and b/.gitbook/assets/RCM Process.png differ diff --git a/.gitbook/assets/RCM.png b/.gitbook/assets/RCM.png new file mode 100644 index 00000000..63afdcfe Binary files /dev/null and b/.gitbook/assets/RCM.png differ diff --git a/.gitbook/assets/Record.jpg b/.gitbook/assets/Record.jpg new file mode 100644 index 00000000..1da2e7cb Binary files /dev/null and b/.gitbook/assets/Record.jpg differ diff --git a/.gitbook/assets/Rules.jpg b/.gitbook/assets/Rules.jpg new file mode 100644 index 00000000..4b45af73 Binary files /dev/null and b/.gitbook/assets/Rules.jpg differ diff --git a/.gitbook/assets/Schema.jpg b/.gitbook/assets/Schema.jpg new file mode 100644 index 00000000..8761a86c Binary files /dev/null and b/.gitbook/assets/Schema.jpg differ diff --git a/.gitbook/assets/Screen Shot 2019-05-29 at 6.44.50 PM.png b/.gitbook/assets/Screen Shot 2019-05-29 at 6.44.50 PM.png new file mode 100644 index 00000000..4f2e17d3 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-05-29 at 6.44.50 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-07-17 at 10.55.06 AM.png b/.gitbook/assets/Screen Shot 2019-07-17 at 10.55.06 AM.png new file mode 100644 index 00000000..efb05ea6 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-07-17 at 10.55.06 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-08-27 at 10.57.30 PM.png b/.gitbook/assets/Screen Shot 2019-08-27 at 10.57.30 PM.png new file mode 100644 index 00000000..7542d0b8 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-08-27 at 10.57.30 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-03 at 10.51.28 AM.png b/.gitbook/assets/Screen Shot 2019-09-03 at 10.51.28 AM.png new file mode 100644 index 00000000..a8ea3aaf Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-03 at 10.51.28 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-03 at 11.13.27 AM.png b/.gitbook/assets/Screen Shot 2019-09-03 at 11.13.27 AM.png new file mode 100644 index 00000000..3db5d52e Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-03 at 11.13.27 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-03 at 11.34.13 AM.png b/.gitbook/assets/Screen Shot 2019-09-03 at 11.34.13 AM.png new file mode 100644 index 00000000..365d26f4 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-03 at 11.34.13 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 1.18.28 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 1.18.28 PM.png new file mode 100644 index 00000000..9d293d1b Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 1.18.28 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 1.29.35 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 1.29.35 PM.png new file mode 100644 index 00000000..3167680f Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 1.29.35 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 1.29.54 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 1.29.54 PM.png new file mode 100644 index 00000000..adcc1b02 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 1.29.54 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 1.30.07 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 1.30.07 PM.png new file mode 100644 index 00000000..657bb17e Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 1.30.07 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 1.32.15 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 1.32.15 PM.png new file mode 100644 index 00000000..2dd4278e Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 1.32.15 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 12.35.34 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 12.35.34 PM.png new file mode 100644 index 00000000..cdf5f45b Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 12.35.34 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 12.38.54 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 12.38.54 PM.png new file mode 100644 index 00000000..51394e53 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 12.38.54 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 12.39.17 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 12.39.17 PM.png new file mode 100644 index 00000000..e646eaa8 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 12.39.17 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 12.39.53 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 12.39.53 PM.png new file mode 100644 index 00000000..a256cd2a Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 12.39.53 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 12.46.02 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 12.46.02 PM.png new file mode 100644 index 00000000..30c7dee5 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 12.46.02 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 12.54.27 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 12.54.27 PM.png new file mode 100644 index 00000000..e3694c7a Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 12.54.27 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-04 at 12.57.24 PM.png b/.gitbook/assets/Screen Shot 2019-09-04 at 12.57.24 PM.png new file mode 100644 index 00000000..d622a976 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-04 at 12.57.24 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-05 at 10.21.44 PM.png b/.gitbook/assets/Screen Shot 2019-09-05 at 10.21.44 PM.png new file mode 100644 index 00000000..7b8f1db0 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-05 at 10.21.44 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-05 at 11.07.43 PM.png b/.gitbook/assets/Screen Shot 2019-09-05 at 11.07.43 PM.png new file mode 100644 index 00000000..669d3cd7 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-05 at 11.07.43 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-09-05 at 11.08.43 PM.png b/.gitbook/assets/Screen Shot 2019-09-05 at 11.08.43 PM.png new file mode 100644 index 00000000..9ce23672 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-09-05 at 11.08.43 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-10-01 at 8.40.33 PM.png b/.gitbook/assets/Screen Shot 2019-10-01 at 8.40.33 PM.png new file mode 100644 index 00000000..8ebfd2d4 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-10-01 at 8.40.33 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-10-01 at 8.50.39 PM.png b/.gitbook/assets/Screen Shot 2019-10-01 at 8.50.39 PM.png new file mode 100644 index 00000000..1c1e8e89 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-10-01 at 8.50.39 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-10-09 at 9.40.55 AM.png b/.gitbook/assets/Screen Shot 2019-10-09 at 9.40.55 AM.png new file mode 100644 index 00000000..110ab382 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-10-09 at 9.40.55 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-10-17 at 4.38.04 PM.png b/.gitbook/assets/Screen Shot 2019-10-17 at 4.38.04 PM.png new file mode 100644 index 00000000..d1901bf4 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-10-17 at 4.38.04 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-11-04 at 2.02.12 PM.png b/.gitbook/assets/Screen Shot 2019-11-04 at 2.02.12 PM.png new file mode 100644 index 00000000..77482255 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-11-04 at 2.02.12 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-11-04 at 2.06.50 PM.png b/.gitbook/assets/Screen Shot 2019-11-04 at 2.06.50 PM.png new file mode 100644 index 00000000..37645b57 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-11-04 at 2.06.50 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-11-04 at 2.10.11 PM (1).png b/.gitbook/assets/Screen Shot 2019-11-04 at 2.10.11 PM (1).png new file mode 100644 index 00000000..829a485d Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-11-04 at 2.10.11 PM (1).png differ diff --git a/.gitbook/assets/Screen Shot 2019-11-18 at 2.52.12 AM.png b/.gitbook/assets/Screen Shot 2019-11-18 at 2.52.12 AM.png new file mode 100644 index 00000000..74f32b4b Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-11-18 at 2.52.12 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-11-18 at 3.01.28 AM.png b/.gitbook/assets/Screen Shot 2019-11-18 at 3.01.28 AM.png new file mode 100644 index 00000000..edbb93d3 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-11-18 at 3.01.28 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-16 at 8.48.34 PM.png b/.gitbook/assets/Screen Shot 2019-12-16 at 8.48.34 PM.png new file mode 100644 index 00000000..8c811819 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-16 at 8.48.34 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-16 at 9.01.13 PM.png b/.gitbook/assets/Screen Shot 2019-12-16 at 9.01.13 PM.png new file mode 100644 index 00000000..36d70f2d Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-16 at 9.01.13 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-16 at 9.20.04 PM.png b/.gitbook/assets/Screen Shot 2019-12-16 at 9.20.04 PM.png new file mode 100644 index 00000000..f5019bcf Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-16 at 9.20.04 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-17 at 10.35.39 AM.png b/.gitbook/assets/Screen Shot 2019-12-17 at 10.35.39 AM.png new file mode 100644 index 00000000..6c2047bc Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-17 at 10.35.39 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-17 at 10.36.54 AM.png b/.gitbook/assets/Screen Shot 2019-12-17 at 10.36.54 AM.png new file mode 100644 index 00000000..0cd3a5be Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-17 at 10.36.54 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-17 at 10.48.31 AM.png b/.gitbook/assets/Screen Shot 2019-12-17 at 10.48.31 AM.png new file mode 100644 index 00000000..b60569e7 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-17 at 10.48.31 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-17 at 10.48.48 AM.png b/.gitbook/assets/Screen Shot 2019-12-17 at 10.48.48 AM.png new file mode 100644 index 00000000..eeb6117f Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-17 at 10.48.48 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-17 at 9.02.51 PM.png b/.gitbook/assets/Screen Shot 2019-12-17 at 9.02.51 PM.png new file mode 100644 index 00000000..7cb67c52 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-17 at 9.02.51 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-17 at 9.03.18 PM.png b/.gitbook/assets/Screen Shot 2019-12-17 at 9.03.18 PM.png new file mode 100644 index 00000000..30738970 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-17 at 9.03.18 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-18 at 1.18.39 PM.png b/.gitbook/assets/Screen Shot 2019-12-18 at 1.18.39 PM.png new file mode 100644 index 00000000..265f669f Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-18 at 1.18.39 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-18 at 1.47.16 PM.png b/.gitbook/assets/Screen Shot 2019-12-18 at 1.47.16 PM.png new file mode 100644 index 00000000..decc2b33 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-18 at 1.47.16 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-18 at 1.48.13 PM.png b/.gitbook/assets/Screen Shot 2019-12-18 at 1.48.13 PM.png new file mode 100644 index 00000000..d67b79c9 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-18 at 1.48.13 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-18 at 1.50.34 PM.png b/.gitbook/assets/Screen Shot 2019-12-18 at 1.50.34 PM.png new file mode 100644 index 00000000..9135e327 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-18 at 1.50.34 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-18 at 12.34.45 PM.png b/.gitbook/assets/Screen Shot 2019-12-18 at 12.34.45 PM.png new file mode 100644 index 00000000..bda7c2e9 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-18 at 12.34.45 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-18 at 12.55.28 PM.png b/.gitbook/assets/Screen Shot 2019-12-18 at 12.55.28 PM.png new file mode 100644 index 00000000..0b9cda28 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-18 at 12.55.28 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-24 at 9.07.59 AM.png b/.gitbook/assets/Screen Shot 2019-12-24 at 9.07.59 AM.png new file mode 100644 index 00000000..c0996bd2 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-24 at 9.07.59 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-24 at 9.08.27 AM.png b/.gitbook/assets/Screen Shot 2019-12-24 at 9.08.27 AM.png new file mode 100644 index 00000000..dea636b8 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-24 at 9.08.27 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-31 at 12.23.52 PM.png b/.gitbook/assets/Screen Shot 2019-12-31 at 12.23.52 PM.png new file mode 100644 index 00000000..b26cee1d Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-31 at 12.23.52 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-31 at 12.31.52 PM.png b/.gitbook/assets/Screen Shot 2019-12-31 at 12.31.52 PM.png new file mode 100644 index 00000000..98643465 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-31 at 12.31.52 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2019-12-31 at 12.32.41 PM.png b/.gitbook/assets/Screen Shot 2019-12-31 at 12.32.41 PM.png new file mode 100644 index 00000000..4e7b667e Binary files /dev/null and b/.gitbook/assets/Screen Shot 2019-12-31 at 12.32.41 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-01-10 at 8.09.09 PM.png b/.gitbook/assets/Screen Shot 2020-01-10 at 8.09.09 PM.png new file mode 100644 index 00000000..8533fbee Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-01-10 at 8.09.09 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-01-30 at 7.22.56 PM.png b/.gitbook/assets/Screen Shot 2020-01-30 at 7.22.56 PM.png new file mode 100644 index 00000000..cc67279f Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-01-30 at 7.22.56 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-01-30 at 7.24.32 PM.png b/.gitbook/assets/Screen Shot 2020-01-30 at 7.24.32 PM.png new file mode 100644 index 00000000..4c4f3b4d Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-01-30 at 7.24.32 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-03-19 at 5.55.49 PM.png b/.gitbook/assets/Screen Shot 2020-03-19 at 5.55.49 PM.png new file mode 100644 index 00000000..692be4bc Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-03-19 at 5.55.49 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-03-20 at 10.57.55 AM.png b/.gitbook/assets/Screen Shot 2020-03-20 at 10.57.55 AM.png new file mode 100644 index 00000000..eb3065e9 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-03-20 at 10.57.55 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-03-20 at 11.17.48 AM.png b/.gitbook/assets/Screen Shot 2020-03-20 at 11.17.48 AM.png new file mode 100644 index 00000000..7be59b1c Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-03-20 at 11.17.48 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-03-30 at 2.35.16 PM.png b/.gitbook/assets/Screen Shot 2020-03-30 at 2.35.16 PM.png new file mode 100644 index 00000000..c2dc99ed Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-03-30 at 2.35.16 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 6.49.40 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 6.49.40 PM.png new file mode 100644 index 00000000..94fc7af7 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 6.49.40 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 6.51.12 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 6.51.12 PM.png new file mode 100644 index 00000000..f3d35af8 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 6.51.12 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 6.53.09 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 6.53.09 PM.png new file mode 100644 index 00000000..664b4961 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 6.53.09 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 6.53.53 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 6.53.53 PM.png new file mode 100644 index 00000000..812cfdc7 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 6.53.53 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 6.55.18 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 6.55.18 PM.png new file mode 100644 index 00000000..0f4e56fb Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 6.55.18 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 6.59.08 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 6.59.08 PM.png new file mode 100644 index 00000000..aa9a7f67 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 6.59.08 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.02.13 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.02.13 PM.png new file mode 100644 index 00000000..829dace8 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.02.13 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.04.56 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.04.56 PM.png new file mode 100644 index 00000000..913741b5 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.04.56 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.07.24 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.07.24 PM.png new file mode 100644 index 00000000..cc90bc6a Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.07.24 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.12.13 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.12.13 PM.png new file mode 100644 index 00000000..817d94b7 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.12.13 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.13.49 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.13.49 PM.png new file mode 100644 index 00000000..a02e7bde Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.13.49 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.15.19 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.15.19 PM.png new file mode 100644 index 00000000..eff658a0 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.15.19 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.17.54 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.17.54 PM.png new file mode 100644 index 00000000..64b4028c Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.17.54 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.19.25 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.19.25 PM.png new file mode 100644 index 00000000..ac6b4967 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.19.25 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.21.11 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.21.11 PM.png new file mode 100644 index 00000000..d59b8ffb Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.21.11 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.22.31 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.22.31 PM.png new file mode 100644 index 00000000..9fa44d93 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.22.31 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.24.45 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.24.45 PM.png new file mode 100644 index 00000000..04eb7b25 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.24.45 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.28.25 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.28.25 PM.png new file mode 100644 index 00000000..bd50e876 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.28.25 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.33.16 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.33.16 PM.png new file mode 100644 index 00000000..c084f915 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.33.16 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 7.57.29 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 7.57.29 PM.png new file mode 100644 index 00000000..27c32d91 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 7.57.29 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 8.02.53 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 8.02.53 PM.png new file mode 100644 index 00000000..9e7d69c8 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 8.02.53 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 8.17.14 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 8.17.14 PM.png new file mode 100644 index 00000000..42ef50e8 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 8.17.14 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 8.17.43 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 8.17.43 PM.png new file mode 100644 index 00000000..6e13b1af Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 8.17.43 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 8.33.44 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 8.33.44 PM.png new file mode 100644 index 00000000..5c4ac9d1 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 8.33.44 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-07 at 8.37.37 PM.png b/.gitbook/assets/Screen Shot 2020-05-07 at 8.37.37 PM.png new file mode 100644 index 00000000..ea5c03a2 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-07 at 8.37.37 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-09 at 11.00.20 PM.png b/.gitbook/assets/Screen Shot 2020-05-09 at 11.00.20 PM.png new file mode 100644 index 00000000..f67a8dec Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-09 at 11.00.20 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-09 at 11.02.15 PM.png b/.gitbook/assets/Screen Shot 2020-05-09 at 11.02.15 PM.png new file mode 100644 index 00000000..d5e1291c Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-09 at 11.02.15 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-09 at 11.04.23 PM.png b/.gitbook/assets/Screen Shot 2020-05-09 at 11.04.23 PM.png new file mode 100644 index 00000000..e30f5ef8 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-09 at 11.04.23 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-09 at 11.06.25 PM.png b/.gitbook/assets/Screen Shot 2020-05-09 at 11.06.25 PM.png new file mode 100644 index 00000000..0048615b Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-09 at 11.06.25 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-05-09 at 11.07.54 PM.png b/.gitbook/assets/Screen Shot 2020-05-09 at 11.07.54 PM.png new file mode 100644 index 00000000..9e5343ab Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-05-09 at 11.07.54 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 5.02.01 AM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 5.02.01 AM.png new file mode 100644 index 00000000..6b362e7e Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 5.02.01 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 5.03.32 AM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 5.03.32 AM.png new file mode 100644 index 00000000..592f5266 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 5.03.32 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 5.06.59 AM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 5.06.59 AM.png new file mode 100644 index 00000000..96d4b4d3 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 5.06.59 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 5.08.15 AM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 5.08.15 AM.png new file mode 100644 index 00000000..439ac308 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 5.08.15 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 5.12.22 AM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 5.12.22 AM.png new file mode 100644 index 00000000..8ff33531 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 5.12.22 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 5.16.11 AM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 5.16.11 AM.png new file mode 100644 index 00000000..9feb5a2f Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 5.16.11 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 5.27.41 AM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 5.27.41 AM.png new file mode 100644 index 00000000..d9b63636 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 5.27.41 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 5.27.51 AM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 5.27.51 AM.png new file mode 100644 index 00000000..41ebd979 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 5.27.51 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 5.29.04 AM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 5.29.04 AM.png new file mode 100644 index 00000000..f462a495 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 5.29.04 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 5.37.54 AM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 5.37.54 AM.png new file mode 100644 index 00000000..25ae5b8b Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 5.37.54 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 5.38.48 AM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 5.38.48 AM.png new file mode 100644 index 00000000..2146d851 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 5.38.48 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 8.17.02 PM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 8.17.02 PM.png new file mode 100644 index 00000000..70ad8860 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 8.17.02 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 8.18.04 PM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 8.18.04 PM.png new file mode 100644 index 00000000..71f73ce5 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 8.18.04 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 8.18.40 PM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 8.18.40 PM.png new file mode 100644 index 00000000..83351b12 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 8.18.40 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 8.19.14 PM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 8.19.14 PM.png new file mode 100644 index 00000000..2aa1dce0 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 8.19.14 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 8.20.18 PM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 8.20.18 PM.png new file mode 100644 index 00000000..368ce500 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 8.20.18 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 8.20.33 PM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 8.20.33 PM.png new file mode 100644 index 00000000..20690e6c Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 8.20.33 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 8.36.20 PM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 8.36.20 PM.png new file mode 100644 index 00000000..d911da25 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 8.36.20 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 8.38.05 PM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 8.38.05 PM.png new file mode 100644 index 00000000..49a9a58a Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 8.38.05 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 9.37.17 PM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 9.37.17 PM.png new file mode 100644 index 00000000..ec8aab87 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 9.37.17 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-07 at 9.43.19 PM.png b/.gitbook/assets/Screen Shot 2020-07-07 at 9.43.19 PM.png new file mode 100644 index 00000000..ded6851d Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-07 at 9.43.19 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-08 at 12.28.39 PM.png b/.gitbook/assets/Screen Shot 2020-07-08 at 12.28.39 PM.png new file mode 100644 index 00000000..e922e6a5 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-08 at 12.28.39 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-08 at 12.37.10 AM.png b/.gitbook/assets/Screen Shot 2020-07-08 at 12.37.10 AM.png new file mode 100644 index 00000000..d834a2d5 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-08 at 12.37.10 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-08 at 12.39.13 AM.png b/.gitbook/assets/Screen Shot 2020-07-08 at 12.39.13 AM.png new file mode 100644 index 00000000..dfa37f36 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-08 at 12.39.13 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-08 at 12.45.19 AM.png b/.gitbook/assets/Screen Shot 2020-07-08 at 12.45.19 AM.png new file mode 100644 index 00000000..33aa086d Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-08 at 12.45.19 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-27 at 8.28.09 PM.png b/.gitbook/assets/Screen Shot 2020-07-27 at 8.28.09 PM.png new file mode 100644 index 00000000..5977b4e9 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-27 at 8.28.09 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-27 at 8.30.57 PM.png b/.gitbook/assets/Screen Shot 2020-07-27 at 8.30.57 PM.png new file mode 100644 index 00000000..ce7ff8df Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-27 at 8.30.57 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-27 at 8.32.20 PM.png b/.gitbook/assets/Screen Shot 2020-07-27 at 8.32.20 PM.png new file mode 100644 index 00000000..e1866ecd Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-27 at 8.32.20 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-27 at 8.37.09 PM.png b/.gitbook/assets/Screen Shot 2020-07-27 at 8.37.09 PM.png new file mode 100644 index 00000000..bc44f8a6 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-27 at 8.37.09 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-27 at 9.09.04 PM.png b/.gitbook/assets/Screen Shot 2020-07-27 at 9.09.04 PM.png new file mode 100644 index 00000000..bca34c7c Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-27 at 9.09.04 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-27 at 9.14.20 PM.png b/.gitbook/assets/Screen Shot 2020-07-27 at 9.14.20 PM.png new file mode 100644 index 00000000..ff75e4cf Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-27 at 9.14.20 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-07-31 at 3.59.32 PM.png b/.gitbook/assets/Screen Shot 2020-07-31 at 3.59.32 PM.png new file mode 100644 index 00000000..fbae54e5 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-07-31 at 3.59.32 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-08-01 at 10.03.07 AM.png b/.gitbook/assets/Screen Shot 2020-08-01 at 10.03.07 AM.png new file mode 100644 index 00000000..fb246c3d Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-08-01 at 10.03.07 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-08-01 at 10.09.20 AM.png b/.gitbook/assets/Screen Shot 2020-08-01 at 10.09.20 AM.png new file mode 100644 index 00000000..e3ec83d5 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-08-01 at 10.09.20 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-08-01 at 10.10.45 AM.png b/.gitbook/assets/Screen Shot 2020-08-01 at 10.10.45 AM.png new file mode 100644 index 00000000..f49b68a7 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-08-01 at 10.10.45 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-08-01 at 10.11.41 AM.png b/.gitbook/assets/Screen Shot 2020-08-01 at 10.11.41 AM.png new file mode 100644 index 00000000..0cce575a Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-08-01 at 10.11.41 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-08-02 at 2.17.34 PM.png b/.gitbook/assets/Screen Shot 2020-08-02 at 2.17.34 PM.png new file mode 100644 index 00000000..ce37ff72 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-08-02 at 2.17.34 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-12-01 at 10.31.09 AM.png b/.gitbook/assets/Screen Shot 2020-12-01 at 10.31.09 AM.png new file mode 100644 index 00000000..b7703aa5 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-12-01 at 10.31.09 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-12-28 at 1.08.45 PM.png b/.gitbook/assets/Screen Shot 2020-12-28 at 1.08.45 PM.png new file mode 100644 index 00000000..d2609962 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-12-28 at 1.08.45 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-12-28 at 1.12.17 PM.png b/.gitbook/assets/Screen Shot 2020-12-28 at 1.12.17 PM.png new file mode 100644 index 00000000..8618b064 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-12-28 at 1.12.17 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-12-28 at 1.13.35 PM.png b/.gitbook/assets/Screen Shot 2020-12-28 at 1.13.35 PM.png new file mode 100644 index 00000000..c0700e73 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-12-28 at 1.13.35 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-12-28 at 1.17.36 PM.png b/.gitbook/assets/Screen Shot 2020-12-28 at 1.17.36 PM.png new file mode 100644 index 00000000..246b4585 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-12-28 at 1.17.36 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-12-28 at 1.20.48 PM.png b/.gitbook/assets/Screen Shot 2020-12-28 at 1.20.48 PM.png new file mode 100644 index 00000000..245d171d Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-12-28 at 1.20.48 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-12-28 at 1.21.42 PM.png b/.gitbook/assets/Screen Shot 2020-12-28 at 1.21.42 PM.png new file mode 100644 index 00000000..5c5341bc Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-12-28 at 1.21.42 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-12-28 at 1.26.01 PM.png b/.gitbook/assets/Screen Shot 2020-12-28 at 1.26.01 PM.png new file mode 100644 index 00000000..d53edefc Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-12-28 at 1.26.01 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2020-12-28 at 1.28.45 PM.png b/.gitbook/assets/Screen Shot 2020-12-28 at 1.28.45 PM.png new file mode 100644 index 00000000..b05ff1a3 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2020-12-28 at 1.28.45 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-02-10 at 8.53.04 PM.png b/.gitbook/assets/Screen Shot 2021-02-10 at 8.53.04 PM.png new file mode 100644 index 00000000..a41e3d30 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-02-10 at 8.53.04 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-02-10 at 8.55.23 PM.png b/.gitbook/assets/Screen Shot 2021-02-10 at 8.55.23 PM.png new file mode 100644 index 00000000..e8da0879 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-02-10 at 8.55.23 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-03-29 at 5.11.02 PM.png b/.gitbook/assets/Screen Shot 2021-03-29 at 5.11.02 PM.png new file mode 100644 index 00000000..713f5da6 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-03-29 at 5.11.02 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-03 at 7.02.23 PM.png b/.gitbook/assets/Screen Shot 2021-04-03 at 7.02.23 PM.png new file mode 100644 index 00000000..97741f34 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-03 at 7.02.23 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-13 at 4.23.32 PM.png b/.gitbook/assets/Screen Shot 2021-04-13 at 4.23.32 PM.png new file mode 100644 index 00000000..46a78744 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-13 at 4.23.32 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-13 at 4.30.29 PM.png b/.gitbook/assets/Screen Shot 2021-04-13 at 4.30.29 PM.png new file mode 100644 index 00000000..29a01392 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-13 at 4.30.29 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-13 at 4.30.36 PM.png b/.gitbook/assets/Screen Shot 2021-04-13 at 4.30.36 PM.png new file mode 100644 index 00000000..9e51ce92 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-13 at 4.30.36 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-13 at 4.34.48 PM.png b/.gitbook/assets/Screen Shot 2021-04-13 at 4.34.48 PM.png new file mode 100644 index 00000000..c0e60124 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-13 at 4.34.48 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-13 at 4.38.15 PM.png b/.gitbook/assets/Screen Shot 2021-04-13 at 4.38.15 PM.png new file mode 100644 index 00000000..252d3603 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-13 at 4.38.15 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-13 at 4.38.21 PM.png b/.gitbook/assets/Screen Shot 2021-04-13 at 4.38.21 PM.png new file mode 100644 index 00000000..623a4935 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-13 at 4.38.21 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-26 at 10.02.12 AM.png b/.gitbook/assets/Screen Shot 2021-04-26 at 10.02.12 AM.png new file mode 100644 index 00000000..0e8449fa Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-26 at 10.02.12 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-26 at 10.07.54 AM.png b/.gitbook/assets/Screen Shot 2021-04-26 at 10.07.54 AM.png new file mode 100644 index 00000000..f4768c0c Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-26 at 10.07.54 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-26 at 10.13.18 AM.png b/.gitbook/assets/Screen Shot 2021-04-26 at 10.13.18 AM.png new file mode 100644 index 00000000..4a7dba86 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-26 at 10.13.18 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-27 at 8.07.58 AM.png b/.gitbook/assets/Screen Shot 2021-04-27 at 8.07.58 AM.png new file mode 100644 index 00000000..f9873917 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-27 at 8.07.58 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-04-27 at 8.14.28 AM.png b/.gitbook/assets/Screen Shot 2021-04-27 at 8.14.28 AM.png new file mode 100644 index 00000000..5e84baf5 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-04-27 at 8.14.28 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-05-10 at 2.24.51 PM.png b/.gitbook/assets/Screen Shot 2021-05-10 at 2.24.51 PM.png new file mode 100644 index 00000000..56ad1893 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-05-10 at 2.24.51 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-07-06 at 9.33.58 AM.png b/.gitbook/assets/Screen Shot 2021-07-06 at 9.33.58 AM.png new file mode 100644 index 00000000..20b1d189 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-07-06 at 9.33.58 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-07-22 at 1.54.04 PM.png b/.gitbook/assets/Screen Shot 2021-07-22 at 1.54.04 PM.png new file mode 100644 index 00000000..86ae8719 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-07-22 at 1.54.04 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-07-22 at 1.57.29 PM.png b/.gitbook/assets/Screen Shot 2021-07-22 at 1.57.29 PM.png new file mode 100644 index 00000000..350704d1 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-07-22 at 1.57.29 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-07-28 at 8.02.40 AM.png b/.gitbook/assets/Screen Shot 2021-07-28 at 8.02.40 AM.png new file mode 100644 index 00000000..7b8c7324 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-07-28 at 8.02.40 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-08-02 at 5.25.08 PM.png b/.gitbook/assets/Screen Shot 2021-08-02 at 5.25.08 PM.png new file mode 100644 index 00000000..00fded59 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-08-02 at 5.25.08 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-08-03 at 9.05.13 AM.png b/.gitbook/assets/Screen Shot 2021-08-03 at 9.05.13 AM.png new file mode 100644 index 00000000..c008ffde Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-08-03 at 9.05.13 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-08-11 at 9.46.48 PM.png b/.gitbook/assets/Screen Shot 2021-08-11 at 9.46.48 PM.png new file mode 100644 index 00000000..0ec2f257 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-08-11 at 9.46.48 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-08-12 at 4.04.53 PM.png b/.gitbook/assets/Screen Shot 2021-08-12 at 4.04.53 PM.png new file mode 100644 index 00000000..908fe05c Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-08-12 at 4.04.53 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-08-12 at 5.26.43 PM.png b/.gitbook/assets/Screen Shot 2021-08-12 at 5.26.43 PM.png new file mode 100644 index 00000000..9ff43a89 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-08-12 at 5.26.43 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-08-12 at 8.03.07 AM.png b/.gitbook/assets/Screen Shot 2021-08-12 at 8.03.07 AM.png new file mode 100644 index 00000000..9fae6ee0 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-08-12 at 8.03.07 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-08-25 at 10.07.43 AM.png b/.gitbook/assets/Screen Shot 2021-08-25 at 10.07.43 AM.png new file mode 100644 index 00000000..e3cedc28 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-08-25 at 10.07.43 AM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-09-15 at 1.11.06 PM.png b/.gitbook/assets/Screen Shot 2021-09-15 at 1.11.06 PM.png new file mode 100644 index 00000000..00e72ee2 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-09-15 at 1.11.06 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-09-15 at 1.14.42 PM.png b/.gitbook/assets/Screen Shot 2021-09-15 at 1.14.42 PM.png new file mode 100644 index 00000000..b47086d7 Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-09-15 at 1.14.42 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-09-15 at 4.17.33 PM.png b/.gitbook/assets/Screen Shot 2021-09-15 at 4.17.33 PM.png new file mode 100644 index 00000000..7761cdaf Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-09-15 at 4.17.33 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-09-15 at 4.32.09 PM.png b/.gitbook/assets/Screen Shot 2021-09-15 at 4.32.09 PM.png new file mode 100644 index 00000000..edc3425f Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-09-15 at 4.32.09 PM.png differ diff --git a/.gitbook/assets/Screen Shot 2021-09-15 at 5.17.14 PM.png b/.gitbook/assets/Screen Shot 2021-09-15 at 5.17.14 PM.png new file mode 100644 index 00000000..bbeb7ddf Binary files /dev/null and b/.gitbook/assets/Screen Shot 2021-09-15 at 5.17.14 PM.png differ diff --git a/.gitbook/assets/Screenshot 2020-07-18 at 1.47.01 PM (1).png b/.gitbook/assets/Screenshot 2020-07-18 at 1.47.01 PM (1).png new file mode 100644 index 00000000..dab60691 Binary files /dev/null and b/.gitbook/assets/Screenshot 2020-07-18 at 1.47.01 PM (1).png differ diff --git a/.gitbook/assets/Screenshot 2020-07-18 at 1.47.01 PM (2).png b/.gitbook/assets/Screenshot 2020-07-18 at 1.47.01 PM (2).png new file mode 100644 index 00000000..c9416bda Binary files /dev/null and b/.gitbook/assets/Screenshot 2020-07-18 at 1.47.01 PM (2).png differ diff --git a/.gitbook/assets/Screenshot 2020-07-18 at 1.47.01 PM.png b/.gitbook/assets/Screenshot 2020-07-18 at 1.47.01 PM.png new file mode 100644 index 00000000..7301c9e6 Binary files /dev/null and b/.gitbook/assets/Screenshot 2020-07-18 at 1.47.01 PM.png differ diff --git a/.gitbook/assets/Screenshot 2020-07-18 at 10.42.49 PM.png b/.gitbook/assets/Screenshot 2020-07-18 at 10.42.49 PM.png new file mode 100644 index 00000000..13296ffc Binary files /dev/null and b/.gitbook/assets/Screenshot 2020-07-18 at 10.42.49 PM.png differ diff --git a/.gitbook/assets/Screenshot 2020-07-19 at 7.31.37 PM.png b/.gitbook/assets/Screenshot 2020-07-19 at 7.31.37 PM.png new file mode 100644 index 00000000..3065272d Binary files /dev/null and b/.gitbook/assets/Screenshot 2020-07-19 at 7.31.37 PM.png differ diff --git a/.gitbook/assets/Screenshot 2020-07-19 at 8.06.15 AM.png b/.gitbook/assets/Screenshot 2020-07-19 at 8.06.15 AM.png new file mode 100644 index 00000000..af347c64 Binary files /dev/null and b/.gitbook/assets/Screenshot 2020-07-19 at 8.06.15 AM.png differ diff --git a/.gitbook/assets/Screenshot 2021-02-17 at 10.53.19 AM.png b/.gitbook/assets/Screenshot 2021-02-17 at 10.53.19 AM.png new file mode 100644 index 00000000..cf649e2f Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-02-17 at 10.53.19 AM.png differ diff --git a/.gitbook/assets/Screenshot 2021-02-17 at 10.57.39 AM.png b/.gitbook/assets/Screenshot 2021-02-17 at 10.57.39 AM.png new file mode 100644 index 00000000..9afb52dd Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-02-17 at 10.57.39 AM.png differ diff --git a/.gitbook/assets/Screenshot 2021-02-17 at 11.00.48 AM.png b/.gitbook/assets/Screenshot 2021-02-17 at 11.00.48 AM.png new file mode 100644 index 00000000..3a53cfb4 Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-02-17 at 11.00.48 AM.png differ diff --git a/.gitbook/assets/Screenshot 2021-02-17 at 9.47.05 AM.png b/.gitbook/assets/Screenshot 2021-02-17 at 9.47.05 AM.png new file mode 100644 index 00000000..0e43c97e Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-02-17 at 9.47.05 AM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 4.02.00 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 4.02.00 PM.png new file mode 100644 index 00000000..f509742e Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 4.02.00 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 4.17.30 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 4.17.30 PM.png new file mode 100644 index 00000000..44c51f5e Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 4.17.30 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 4.23.08 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 4.23.08 PM.png new file mode 100644 index 00000000..e55624b5 Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 4.23.08 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 4.23.27 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 4.23.27 PM.png new file mode 100644 index 00000000..a93a131e Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 4.23.27 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 4.25.09 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 4.25.09 PM.png new file mode 100644 index 00000000..0b33478c Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 4.25.09 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 4.46.39 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 4.46.39 PM.png new file mode 100644 index 00000000..2ffc2326 Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 4.46.39 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 5.00.59 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 5.00.59 PM.png new file mode 100644 index 00000000..36326cda Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 5.00.59 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 5.02.00 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 5.02.00 PM.png new file mode 100644 index 00000000..d6449fcc Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 5.02.00 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 5.03.40 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 5.03.40 PM.png new file mode 100644 index 00000000..441dbec7 Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 5.03.40 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 5.04.25 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 5.04.25 PM.png new file mode 100644 index 00000000..1d7bcf2e Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 5.04.25 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 5.05.31 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 5.05.31 PM.png new file mode 100644 index 00000000..5b239bcd Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 5.05.31 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 5.27.04 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 5.27.04 PM.png new file mode 100644 index 00000000..90eca121 Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 5.27.04 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-14 at 5.44.53 PM.png b/.gitbook/assets/Screenshot 2021-06-14 at 5.44.53 PM.png new file mode 100644 index 00000000..72a780a0 Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-14 at 5.44.53 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-20 at 9.22.45 PM.png b/.gitbook/assets/Screenshot 2021-06-20 at 9.22.45 PM.png new file mode 100644 index 00000000..5bcce3d3 Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-20 at 9.22.45 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-20 at 9.24.06 PM.png b/.gitbook/assets/Screenshot 2021-06-20 at 9.24.06 PM.png new file mode 100644 index 00000000..2d5135b6 Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-20 at 9.24.06 PM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-21 at 8.38.56 AM.png b/.gitbook/assets/Screenshot 2021-06-21 at 8.38.56 AM.png new file mode 100644 index 00000000..249a5a63 Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-21 at 8.38.56 AM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-21 at 8.42.19 AM.png b/.gitbook/assets/Screenshot 2021-06-21 at 8.42.19 AM.png new file mode 100644 index 00000000..8ce00ee4 Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-21 at 8.42.19 AM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-21 at 9.04.11 AM.png b/.gitbook/assets/Screenshot 2021-06-21 at 9.04.11 AM.png new file mode 100644 index 00000000..b4ddd4a2 Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-21 at 9.04.11 AM.png differ diff --git a/.gitbook/assets/Screenshot 2021-06-21 at 9.05.39 AM.png b/.gitbook/assets/Screenshot 2021-06-21 at 9.05.39 AM.png new file mode 100644 index 00000000..1ab2a0fc Binary files /dev/null and b/.gitbook/assets/Screenshot 2021-06-21 at 9.05.39 AM.png differ diff --git a/.gitbook/assets/Shapes.jpg b/.gitbook/assets/Shapes.jpg new file mode 100644 index 00000000..1837109e Binary files /dev/null and b/.gitbook/assets/Shapes.jpg differ diff --git a/.gitbook/assets/Source.jpg b/.gitbook/assets/Source.jpg new file mode 100644 index 00000000..a017923b Binary files /dev/null and b/.gitbook/assets/Source.jpg differ diff --git a/.gitbook/assets/Statistical Process Control for Data Quality (Animation) Aug 2021.gif b/.gitbook/assets/Statistical Process Control for Data Quality (Animation) Aug 2021.gif new file mode 100644 index 00000000..42d61f93 Binary files /dev/null and b/.gitbook/assets/Statistical Process Control for Data Quality (Animation) Aug 2021.gif differ diff --git a/.gitbook/assets/adaptive_rules.gif b/.gitbook/assets/adaptive_rules.gif new file mode 100644 index 00000000..fcf42d9f Binary files /dev/null and b/.gitbook/assets/adaptive_rules.gif differ diff --git a/.gitbook/assets/adding-data-concepts.gif b/.gitbook/assets/adding-data-concepts.gif new file mode 100644 index 00000000..0f68dce3 Binary files /dev/null and b/.gitbook/assets/adding-data-concepts.gif differ diff --git a/.gitbook/assets/agentTenant.png b/.gitbook/assets/agentTenant.png new file mode 100644 index 00000000..f9751274 Binary files /dev/null and b/.gitbook/assets/agentTenant.png differ diff --git a/.gitbook/assets/agent_shuffle.gif b/.gitbook/assets/agent_shuffle.gif new file mode 100644 index 00000000..d29e8497 Binary files /dev/null and b/.gitbook/assets/agent_shuffle.gif differ diff --git a/.gitbook/assets/agenttenant.png b/.gitbook/assets/agenttenant.png new file mode 100644 index 00000000..f9751274 Binary files /dev/null and b/.gitbook/assets/agenttenant.png differ diff --git a/.gitbook/assets/alerts.gif b/.gitbook/assets/alerts.gif new file mode 100644 index 00000000..07547069 Binary files /dev/null and b/.gitbook/assets/alerts.gif differ diff --git a/.gitbook/assets/assignments (1).gif b/.gitbook/assets/assignments (1).gif new file mode 100644 index 00000000..29036407 Binary files /dev/null and b/.gitbook/assets/assignments (1).gif differ diff --git a/.gitbook/assets/assignments.gif b/.gitbook/assets/assignments.gif new file mode 100644 index 00000000..29036407 Binary files /dev/null and b/.gitbook/assets/assignments.gif differ diff --git a/.gitbook/assets/bank-loan.jpeg b/.gitbook/assets/bank-loan.jpeg new file mode 100644 index 00000000..4e63f2f2 Binary files /dev/null and b/.gitbook/assets/bank-loan.jpeg differ diff --git a/.gitbook/assets/bank-loan1.jpeg b/.gitbook/assets/bank-loan1.jpeg new file mode 100644 index 00000000..1427b84e Binary files /dev/null and b/.gitbook/assets/bank-loan1.jpeg differ diff --git a/.gitbook/assets/behaviors.gif b/.gitbook/assets/behaviors.gif new file mode 100644 index 00000000..e9001542 Binary files /dev/null and b/.gitbook/assets/behaviors.gif differ diff --git a/.gitbook/assets/break_records (1).gif b/.gitbook/assets/break_records (1).gif new file mode 100644 index 00000000..838480d9 Binary files /dev/null and b/.gitbook/assets/break_records (1).gif differ diff --git a/.gitbook/assets/break_records.gif b/.gitbook/assets/break_records.gif new file mode 100644 index 00000000..838480d9 Binary files /dev/null and b/.gitbook/assets/break_records.gif differ diff --git a/.gitbook/assets/bulk_actions.gif b/.gitbook/assets/bulk_actions.gif new file mode 100644 index 00000000..b6855ece Binary files /dev/null and b/.gitbook/assets/bulk_actions.gif differ diff --git a/.gitbook/assets/business_units.gif b/.gitbook/assets/business_units.gif new file mode 100644 index 00000000..34dfc195 Binary files /dev/null and b/.gitbook/assets/business_units.gif differ diff --git a/.gitbook/assets/cdh-jars-hive-jdbc.png b/.gitbook/assets/cdh-jars-hive-jdbc.png new file mode 100644 index 00000000..98432bae Binary files /dev/null and b/.gitbook/assets/cdh-jars-hive-jdbc.png differ diff --git a/.gitbook/assets/colStatRules.png b/.gitbook/assets/colStatRules.png new file mode 100644 index 00000000..04f544c4 Binary files /dev/null and b/.gitbook/assets/colStatRules.png differ diff --git a/.gitbook/assets/completeness_report.gif b/.gitbook/assets/completeness_report.gif new file mode 100644 index 00000000..1342aae7 Binary files /dev/null and b/.gitbook/assets/completeness_report.gif differ diff --git a/.gitbook/assets/connections (1).gif b/.gitbook/assets/connections (1).gif new file mode 100644 index 00000000..c41e76d1 Binary files /dev/null and b/.gitbook/assets/connections (1).gif differ diff --git a/.gitbook/assets/connections (2).gif b/.gitbook/assets/connections (2).gif new file mode 100644 index 00000000..2a6cafbc Binary files /dev/null and b/.gitbook/assets/connections (2).gif differ diff --git a/.gitbook/assets/connections.gif b/.gitbook/assets/connections.gif new file mode 100644 index 00000000..c41e76d1 Binary files /dev/null and b/.gitbook/assets/connections.gif differ diff --git a/.gitbook/assets/creating-semantic-regex (1).gif b/.gitbook/assets/creating-semantic-regex (1).gif new file mode 100644 index 00000000..c3e74a68 Binary files /dev/null and b/.gitbook/assets/creating-semantic-regex (1).gif differ diff --git a/.gitbook/assets/creating-semantic-regex.gif b/.gitbook/assets/creating-semantic-regex.gif new file mode 100644 index 00000000..c3e74a68 Binary files /dev/null and b/.gitbook/assets/creating-semantic-regex.gif differ diff --git a/.gitbook/assets/cyber-anomaly-detection-4.png b/.gitbook/assets/cyber-anomaly-detection-4.png new file mode 100644 index 00000000..c36a93fc Binary files /dev/null and b/.gitbook/assets/cyber-anomaly-detection-4.png differ diff --git a/.gitbook/assets/dataconcept-rerun-job.gif b/.gitbook/assets/dataconcept-rerun-job.gif new file mode 100644 index 00000000..0891ed14 Binary files /dev/null and b/.gitbook/assets/dataconcept-rerun-job.gif differ diff --git a/.gitbook/assets/db-jdbc.png b/.gitbook/assets/db-jdbc.png new file mode 100644 index 00000000..a43c025b Binary files /dev/null and b/.gitbook/assets/db-jdbc.png differ diff --git a/.gitbook/assets/docs.gif b/.gitbook/assets/docs.gif new file mode 100644 index 00000000..963ce972 Binary files /dev/null and b/.gitbook/assets/docs.gif differ diff --git a/.gitbook/assets/dqjobsemanticon (1).gif b/.gitbook/assets/dqjobsemanticon (1).gif new file mode 100644 index 00000000..7c53504e Binary files /dev/null and b/.gitbook/assets/dqjobsemanticon (1).gif differ diff --git a/.gitbook/assets/dqjobsemanticon.gif b/.gitbook/assets/dqjobsemanticon.gif new file mode 100644 index 00000000..7c53504e Binary files /dev/null and b/.gitbook/assets/dqjobsemanticon.gif differ diff --git a/.gitbook/assets/drivers.png b/.gitbook/assets/drivers.png new file mode 100644 index 00000000..7788b3ee Binary files /dev/null and b/.gitbook/assets/drivers.png differ diff --git a/.gitbook/assets/dv-jdbc2.png b/.gitbook/assets/dv-jdbc2.png new file mode 100644 index 00000000..d16a3d3f Binary files /dev/null and b/.gitbook/assets/dv-jdbc2.png differ diff --git a/.gitbook/assets/explorer (1).gif b/.gitbook/assets/explorer (1).gif new file mode 100644 index 00000000..f34eb566 Binary files /dev/null and b/.gitbook/assets/explorer (1).gif differ diff --git a/.gitbook/assets/explorer (2).gif b/.gitbook/assets/explorer (2).gif new file mode 100644 index 00000000..92e5d058 Binary files /dev/null and b/.gitbook/assets/explorer (2).gif differ diff --git a/.gitbook/assets/explorer (3).gif b/.gitbook/assets/explorer (3).gif new file mode 100644 index 00000000..53a52f47 Binary files /dev/null and b/.gitbook/assets/explorer (3).gif differ diff --git a/.gitbook/assets/explorer (4).gif b/.gitbook/assets/explorer (4).gif new file mode 100644 index 00000000..4cd29194 Binary files /dev/null and b/.gitbook/assets/explorer (4).gif differ diff --git a/.gitbook/assets/explorer.gif b/.gitbook/assets/explorer.gif new file mode 100644 index 00000000..f34eb566 Binary files /dev/null and b/.gitbook/assets/explorer.gif differ diff --git a/.gitbook/assets/explorer_create.gif b/.gitbook/assets/explorer_create.gif new file mode 100644 index 00000000..a6a044f6 Binary files /dev/null and b/.gitbook/assets/explorer_create.gif differ diff --git a/.gitbook/assets/full-package-install-packages.png b/.gitbook/assets/full-package-install-packages.png new file mode 100644 index 00000000..f2559327 Binary files /dev/null and b/.gitbook/assets/full-package-install-packages.png differ diff --git a/.gitbook/assets/global-params.png b/.gitbook/assets/global-params.png new file mode 100644 index 00000000..51a5f33e Binary files /dev/null and b/.gitbook/assets/global-params.png differ diff --git a/.gitbook/assets/image (1).png b/.gitbook/assets/image (1).png new file mode 100644 index 00000000..dba3adaa Binary files /dev/null and b/.gitbook/assets/image (1).png differ diff --git a/.gitbook/assets/image (10).png b/.gitbook/assets/image (10).png new file mode 100644 index 00000000..24fb6d1e Binary files /dev/null and b/.gitbook/assets/image (10).png differ diff --git a/.gitbook/assets/image (100).png b/.gitbook/assets/image (100).png new file mode 100644 index 00000000..9275e098 Binary files /dev/null and b/.gitbook/assets/image (100).png differ diff --git a/.gitbook/assets/image (101).png b/.gitbook/assets/image (101).png new file mode 100644 index 00000000..619557af Binary files /dev/null and b/.gitbook/assets/image (101).png differ diff --git a/.gitbook/assets/image (102).png b/.gitbook/assets/image (102).png new file mode 100644 index 00000000..3667f15a Binary files /dev/null and b/.gitbook/assets/image (102).png differ diff --git a/.gitbook/assets/image (103).png b/.gitbook/assets/image (103).png new file mode 100644 index 00000000..622715f7 Binary files /dev/null and b/.gitbook/assets/image (103).png differ diff --git a/.gitbook/assets/image (104).png b/.gitbook/assets/image (104).png new file mode 100644 index 00000000..377f0e7d Binary files /dev/null and b/.gitbook/assets/image (104).png differ diff --git a/.gitbook/assets/image (105).png b/.gitbook/assets/image (105).png new file mode 100644 index 00000000..fb2a4416 Binary files /dev/null and b/.gitbook/assets/image (105).png differ diff --git a/.gitbook/assets/image (106).png b/.gitbook/assets/image (106).png new file mode 100644 index 00000000..2cd267cc Binary files /dev/null and b/.gitbook/assets/image (106).png differ diff --git a/.gitbook/assets/image (107).png b/.gitbook/assets/image (107).png new file mode 100644 index 00000000..35d82626 Binary files /dev/null and b/.gitbook/assets/image (107).png differ diff --git a/.gitbook/assets/image (108).png b/.gitbook/assets/image (108).png new file mode 100644 index 00000000..58b5ea1f Binary files /dev/null and b/.gitbook/assets/image (108).png differ diff --git a/.gitbook/assets/image (109).png b/.gitbook/assets/image (109).png new file mode 100644 index 00000000..0fcfe85e Binary files /dev/null and b/.gitbook/assets/image (109).png differ diff --git a/.gitbook/assets/image (11) (1).png b/.gitbook/assets/image (11) (1).png new file mode 100644 index 00000000..e184952d Binary files /dev/null and b/.gitbook/assets/image (11) (1).png differ diff --git a/.gitbook/assets/image (11).png b/.gitbook/assets/image (11).png new file mode 100644 index 00000000..6170bdb9 Binary files /dev/null and b/.gitbook/assets/image (11).png differ diff --git a/.gitbook/assets/image (110).png b/.gitbook/assets/image (110).png new file mode 100644 index 00000000..4be7bb28 Binary files /dev/null and b/.gitbook/assets/image (110).png differ diff --git a/.gitbook/assets/image (111).png b/.gitbook/assets/image (111).png new file mode 100644 index 00000000..2413a401 Binary files /dev/null and b/.gitbook/assets/image (111).png differ diff --git a/.gitbook/assets/image (112).png b/.gitbook/assets/image (112).png new file mode 100644 index 00000000..2bb6021d Binary files /dev/null and b/.gitbook/assets/image (112).png differ diff --git a/.gitbook/assets/image (113).png b/.gitbook/assets/image (113).png new file mode 100644 index 00000000..37fb1c53 Binary files /dev/null and b/.gitbook/assets/image (113).png differ diff --git a/.gitbook/assets/image (114).png b/.gitbook/assets/image (114).png new file mode 100644 index 00000000..b848b44b Binary files /dev/null and b/.gitbook/assets/image (114).png differ diff --git a/.gitbook/assets/image (115).png b/.gitbook/assets/image (115).png new file mode 100644 index 00000000..4b503ab7 Binary files /dev/null and b/.gitbook/assets/image (115).png differ diff --git a/.gitbook/assets/image (116).png b/.gitbook/assets/image (116).png new file mode 100644 index 00000000..5d793769 Binary files /dev/null and b/.gitbook/assets/image (116).png differ diff --git a/.gitbook/assets/image (117).png b/.gitbook/assets/image (117).png new file mode 100644 index 00000000..15753f3c Binary files /dev/null and b/.gitbook/assets/image (117).png differ diff --git a/.gitbook/assets/image (118).png b/.gitbook/assets/image (118).png new file mode 100644 index 00000000..94649bbf Binary files /dev/null and b/.gitbook/assets/image (118).png differ diff --git a/.gitbook/assets/image (119).png b/.gitbook/assets/image (119).png new file mode 100644 index 00000000..52401605 Binary files /dev/null and b/.gitbook/assets/image (119).png differ diff --git a/.gitbook/assets/image (12).png b/.gitbook/assets/image (12).png new file mode 100644 index 00000000..61893af1 Binary files /dev/null and b/.gitbook/assets/image (12).png differ diff --git a/.gitbook/assets/image (120).png b/.gitbook/assets/image (120).png new file mode 100644 index 00000000..2a87267e Binary files /dev/null and b/.gitbook/assets/image (120).png differ diff --git a/.gitbook/assets/image (121).png b/.gitbook/assets/image (121).png new file mode 100644 index 00000000..a437e978 Binary files /dev/null and b/.gitbook/assets/image (121).png differ diff --git a/.gitbook/assets/image (122).png b/.gitbook/assets/image (122).png new file mode 100644 index 00000000..134c1a2b Binary files /dev/null and b/.gitbook/assets/image (122).png differ diff --git a/.gitbook/assets/image (123).png b/.gitbook/assets/image (123).png new file mode 100644 index 00000000..7a60a339 Binary files /dev/null and b/.gitbook/assets/image (123).png differ diff --git a/.gitbook/assets/image (124).png b/.gitbook/assets/image (124).png new file mode 100644 index 00000000..181eae65 Binary files /dev/null and b/.gitbook/assets/image (124).png differ diff --git a/.gitbook/assets/image (125).png b/.gitbook/assets/image (125).png new file mode 100644 index 00000000..015a7a08 Binary files /dev/null and b/.gitbook/assets/image (125).png differ diff --git a/.gitbook/assets/image (126).png b/.gitbook/assets/image (126).png new file mode 100644 index 00000000..36d8c5da Binary files /dev/null and b/.gitbook/assets/image (126).png differ diff --git a/.gitbook/assets/image (127).png b/.gitbook/assets/image (127).png new file mode 100644 index 00000000..da19e02c Binary files /dev/null and b/.gitbook/assets/image (127).png differ diff --git a/.gitbook/assets/image (128).png b/.gitbook/assets/image (128).png new file mode 100644 index 00000000..da19e02c Binary files /dev/null and b/.gitbook/assets/image (128).png differ diff --git a/.gitbook/assets/image (129).png b/.gitbook/assets/image (129).png new file mode 100644 index 00000000..2413a401 Binary files /dev/null and b/.gitbook/assets/image (129).png differ diff --git a/.gitbook/assets/image (13).png b/.gitbook/assets/image (13).png new file mode 100644 index 00000000..176a001b Binary files /dev/null and b/.gitbook/assets/image (13).png differ diff --git a/.gitbook/assets/image (130).png b/.gitbook/assets/image (130).png new file mode 100644 index 00000000..0b08762c Binary files /dev/null and b/.gitbook/assets/image (130).png differ diff --git a/.gitbook/assets/image (131).png b/.gitbook/assets/image (131).png new file mode 100644 index 00000000..3e874a5d Binary files /dev/null and b/.gitbook/assets/image (131).png differ diff --git a/.gitbook/assets/image (132).png b/.gitbook/assets/image (132).png new file mode 100644 index 00000000..619557af Binary files /dev/null and b/.gitbook/assets/image (132).png differ diff --git a/.gitbook/assets/image (133).png b/.gitbook/assets/image (133).png new file mode 100644 index 00000000..6727a0bc Binary files /dev/null and b/.gitbook/assets/image (133).png differ diff --git a/.gitbook/assets/image (134).png b/.gitbook/assets/image (134).png new file mode 100644 index 00000000..792d9bcc Binary files /dev/null and b/.gitbook/assets/image (134).png differ diff --git a/.gitbook/assets/image (135).png b/.gitbook/assets/image (135).png new file mode 100644 index 00000000..c8897749 Binary files /dev/null and b/.gitbook/assets/image (135).png differ diff --git a/.gitbook/assets/image (136).png b/.gitbook/assets/image (136).png new file mode 100644 index 00000000..a1f0d037 Binary files /dev/null and b/.gitbook/assets/image (136).png differ diff --git a/.gitbook/assets/image (137).png b/.gitbook/assets/image (137).png new file mode 100644 index 00000000..1e4573bf Binary files /dev/null and b/.gitbook/assets/image (137).png differ diff --git a/.gitbook/assets/image (138).png b/.gitbook/assets/image (138).png new file mode 100644 index 00000000..eca84eb1 Binary files /dev/null and b/.gitbook/assets/image (138).png differ diff --git a/.gitbook/assets/image (139).png b/.gitbook/assets/image (139).png new file mode 100644 index 00000000..15514540 Binary files /dev/null and b/.gitbook/assets/image (139).png differ diff --git a/.gitbook/assets/image (14).png b/.gitbook/assets/image (14).png new file mode 100644 index 00000000..3d69c3e2 Binary files /dev/null and b/.gitbook/assets/image (14).png differ diff --git a/.gitbook/assets/image (140).png b/.gitbook/assets/image (140).png new file mode 100644 index 00000000..cf305c73 Binary files /dev/null and b/.gitbook/assets/image (140).png differ diff --git a/.gitbook/assets/image (141).png b/.gitbook/assets/image (141).png new file mode 100644 index 00000000..00a5b050 Binary files /dev/null and b/.gitbook/assets/image (141).png differ diff --git a/.gitbook/assets/image (142).png b/.gitbook/assets/image (142).png new file mode 100644 index 00000000..2a87267e Binary files /dev/null and b/.gitbook/assets/image (142).png differ diff --git a/.gitbook/assets/image (143).png b/.gitbook/assets/image (143).png new file mode 100644 index 00000000..792d9bcc Binary files /dev/null and b/.gitbook/assets/image (143).png differ diff --git a/.gitbook/assets/image (144).png b/.gitbook/assets/image (144).png new file mode 100644 index 00000000..a14372d1 Binary files /dev/null and b/.gitbook/assets/image (144).png differ diff --git a/.gitbook/assets/image (145).png b/.gitbook/assets/image (145).png new file mode 100644 index 00000000..924b169b Binary files /dev/null and b/.gitbook/assets/image (145).png differ diff --git a/.gitbook/assets/image (146).png b/.gitbook/assets/image (146).png new file mode 100644 index 00000000..1bf3e9b0 Binary files /dev/null and b/.gitbook/assets/image (146).png differ diff --git a/.gitbook/assets/image (147).png b/.gitbook/assets/image (147).png new file mode 100644 index 00000000..e829bf7f Binary files /dev/null and b/.gitbook/assets/image (147).png differ diff --git a/.gitbook/assets/image (148).png b/.gitbook/assets/image (148).png new file mode 100644 index 00000000..7a60a339 Binary files /dev/null and b/.gitbook/assets/image (148).png differ diff --git a/.gitbook/assets/image (149).png b/.gitbook/assets/image (149).png new file mode 100644 index 00000000..535b5129 Binary files /dev/null and b/.gitbook/assets/image (149).png differ diff --git a/.gitbook/assets/image (15).png b/.gitbook/assets/image (15).png new file mode 100644 index 00000000..23b99f01 Binary files /dev/null and b/.gitbook/assets/image (15).png differ diff --git a/.gitbook/assets/image (150).png b/.gitbook/assets/image (150).png new file mode 100644 index 00000000..590f8a4e Binary files /dev/null and b/.gitbook/assets/image (150).png differ diff --git a/.gitbook/assets/image (151).png b/.gitbook/assets/image (151).png new file mode 100644 index 00000000..edd365ce Binary files /dev/null and b/.gitbook/assets/image (151).png differ diff --git a/.gitbook/assets/image (16).png b/.gitbook/assets/image (16).png new file mode 100644 index 00000000..c0022250 Binary files /dev/null and b/.gitbook/assets/image (16).png differ diff --git a/.gitbook/assets/image (17).png b/.gitbook/assets/image (17).png new file mode 100644 index 00000000..3278bff4 Binary files /dev/null and b/.gitbook/assets/image (17).png differ diff --git a/.gitbook/assets/image (18) (1).png b/.gitbook/assets/image (18) (1).png new file mode 100644 index 00000000..24fb6d1e Binary files /dev/null and b/.gitbook/assets/image (18) (1).png differ diff --git a/.gitbook/assets/image (18) (2).png b/.gitbook/assets/image (18) (2).png new file mode 100644 index 00000000..24fb6d1e Binary files /dev/null and b/.gitbook/assets/image (18) (2).png differ diff --git a/.gitbook/assets/image (18) (3).png b/.gitbook/assets/image (18) (3).png new file mode 100644 index 00000000..24fb6d1e Binary files /dev/null and b/.gitbook/assets/image (18) (3).png differ diff --git a/.gitbook/assets/image (18) (4).png b/.gitbook/assets/image (18) (4).png new file mode 100644 index 00000000..24fb6d1e Binary files /dev/null and b/.gitbook/assets/image (18) (4).png differ diff --git a/.gitbook/assets/image (18) (5).png b/.gitbook/assets/image (18) (5).png new file mode 100644 index 00000000..24fb6d1e Binary files /dev/null and b/.gitbook/assets/image (18) (5).png differ diff --git a/.gitbook/assets/image (18).png b/.gitbook/assets/image (18).png new file mode 100644 index 00000000..24fb6d1e Binary files /dev/null and b/.gitbook/assets/image (18).png differ diff --git a/.gitbook/assets/image (19).png b/.gitbook/assets/image (19).png new file mode 100644 index 00000000..23b99f01 Binary files /dev/null and b/.gitbook/assets/image (19).png differ diff --git a/.gitbook/assets/image (2) (1).png b/.gitbook/assets/image (2) (1).png new file mode 100644 index 00000000..2ccb2747 Binary files /dev/null and b/.gitbook/assets/image (2) (1).png differ diff --git a/.gitbook/assets/image (2).png b/.gitbook/assets/image (2).png new file mode 100644 index 00000000..60fa6e7a Binary files /dev/null and b/.gitbook/assets/image (2).png differ diff --git a/.gitbook/assets/image (20).png b/.gitbook/assets/image (20).png new file mode 100644 index 00000000..886e7551 Binary files /dev/null and b/.gitbook/assets/image (20).png differ diff --git a/.gitbook/assets/image (21).png b/.gitbook/assets/image (21).png new file mode 100644 index 00000000..e184952d Binary files /dev/null and b/.gitbook/assets/image (21).png differ diff --git a/.gitbook/assets/image (22).png b/.gitbook/assets/image (22).png new file mode 100644 index 00000000..4f03b19f Binary files /dev/null and b/.gitbook/assets/image (22).png differ diff --git a/.gitbook/assets/image (23).png b/.gitbook/assets/image (23).png new file mode 100644 index 00000000..5fd0a35e Binary files /dev/null and b/.gitbook/assets/image (23).png differ diff --git a/.gitbook/assets/image (24).png b/.gitbook/assets/image (24).png new file mode 100644 index 00000000..c41e1ea4 Binary files /dev/null and b/.gitbook/assets/image (24).png differ diff --git a/.gitbook/assets/image (25).png b/.gitbook/assets/image (25).png new file mode 100644 index 00000000..2787cc1d Binary files /dev/null and b/.gitbook/assets/image (25).png differ diff --git a/.gitbook/assets/image (26) (1).png b/.gitbook/assets/image (26) (1).png new file mode 100644 index 00000000..3d69c3e2 Binary files /dev/null and b/.gitbook/assets/image (26) (1).png differ diff --git a/.gitbook/assets/image (26).png b/.gitbook/assets/image (26).png new file mode 100644 index 00000000..3d69c3e2 Binary files /dev/null and b/.gitbook/assets/image (26).png differ diff --git a/.gitbook/assets/image (27).png b/.gitbook/assets/image (27).png new file mode 100644 index 00000000..efe6c53a Binary files /dev/null and b/.gitbook/assets/image (27).png differ diff --git a/.gitbook/assets/image (28).png b/.gitbook/assets/image (28).png new file mode 100644 index 00000000..2a0b1f46 Binary files /dev/null and b/.gitbook/assets/image (28).png differ diff --git a/.gitbook/assets/image (29).png b/.gitbook/assets/image (29).png new file mode 100644 index 00000000..2b249338 Binary files /dev/null and b/.gitbook/assets/image (29).png differ diff --git a/.gitbook/assets/image (3).png b/.gitbook/assets/image (3).png new file mode 100644 index 00000000..b2df8f45 Binary files /dev/null and b/.gitbook/assets/image (3).png differ diff --git a/.gitbook/assets/image (30).png b/.gitbook/assets/image (30).png new file mode 100644 index 00000000..977b5c02 Binary files /dev/null and b/.gitbook/assets/image (30).png differ diff --git a/.gitbook/assets/image (31).png b/.gitbook/assets/image (31).png new file mode 100644 index 00000000..0ee99a14 Binary files /dev/null and b/.gitbook/assets/image (31).png differ diff --git a/.gitbook/assets/image (32).png b/.gitbook/assets/image (32).png new file mode 100644 index 00000000..9bdd4156 Binary files /dev/null and b/.gitbook/assets/image (32).png differ diff --git a/.gitbook/assets/image (33).png b/.gitbook/assets/image (33).png new file mode 100644 index 00000000..69327d86 Binary files /dev/null and b/.gitbook/assets/image (33).png differ diff --git a/.gitbook/assets/image (34).png b/.gitbook/assets/image (34).png new file mode 100644 index 00000000..0e37abe9 Binary files /dev/null and b/.gitbook/assets/image (34).png differ diff --git a/.gitbook/assets/image (35).png b/.gitbook/assets/image (35).png new file mode 100644 index 00000000..bcd01403 Binary files /dev/null and b/.gitbook/assets/image (35).png differ diff --git a/.gitbook/assets/image (36).png b/.gitbook/assets/image (36).png new file mode 100644 index 00000000..0ee99a14 Binary files /dev/null and b/.gitbook/assets/image (36).png differ diff --git a/.gitbook/assets/image (37).png b/.gitbook/assets/image (37).png new file mode 100644 index 00000000..3c08ab23 Binary files /dev/null and b/.gitbook/assets/image (37).png differ diff --git a/.gitbook/assets/image (38).png b/.gitbook/assets/image (38).png new file mode 100644 index 00000000..63e281ff Binary files /dev/null and b/.gitbook/assets/image (38).png differ diff --git a/.gitbook/assets/image (39).png b/.gitbook/assets/image (39).png new file mode 100644 index 00000000..977b5c02 Binary files /dev/null and b/.gitbook/assets/image (39).png differ diff --git a/.gitbook/assets/image (4).png b/.gitbook/assets/image (4).png new file mode 100644 index 00000000..a9a04112 Binary files /dev/null and b/.gitbook/assets/image (4).png differ diff --git a/.gitbook/assets/image (40).png b/.gitbook/assets/image (40).png new file mode 100644 index 00000000..5e724ac5 Binary files /dev/null and b/.gitbook/assets/image (40).png differ diff --git a/.gitbook/assets/image (41).png b/.gitbook/assets/image (41).png new file mode 100644 index 00000000..18a8ac4e Binary files /dev/null and b/.gitbook/assets/image (41).png differ diff --git a/.gitbook/assets/image (42).png b/.gitbook/assets/image (42).png new file mode 100644 index 00000000..81635a39 Binary files /dev/null and b/.gitbook/assets/image (42).png differ diff --git a/.gitbook/assets/image (43).png b/.gitbook/assets/image (43).png new file mode 100644 index 00000000..49830d7e Binary files /dev/null and b/.gitbook/assets/image (43).png differ diff --git a/.gitbook/assets/image (44) (1).png b/.gitbook/assets/image (44) (1).png new file mode 100644 index 00000000..bcd01403 Binary files /dev/null and b/.gitbook/assets/image (44) (1).png differ diff --git a/.gitbook/assets/image (44).png b/.gitbook/assets/image (44).png new file mode 100644 index 00000000..1e04e1e2 Binary files /dev/null and b/.gitbook/assets/image (44).png differ diff --git a/.gitbook/assets/image (45).png b/.gitbook/assets/image (45).png new file mode 100644 index 00000000..9c2dfec1 Binary files /dev/null and b/.gitbook/assets/image (45).png differ diff --git a/.gitbook/assets/image (46).png b/.gitbook/assets/image (46).png new file mode 100644 index 00000000..6f285491 Binary files /dev/null and b/.gitbook/assets/image (46).png differ diff --git a/.gitbook/assets/image (47) (1).png b/.gitbook/assets/image (47) (1).png new file mode 100644 index 00000000..3c08ab23 Binary files /dev/null and b/.gitbook/assets/image (47) (1).png differ diff --git a/.gitbook/assets/image (47).png b/.gitbook/assets/image (47).png new file mode 100644 index 00000000..6d473e17 Binary files /dev/null and b/.gitbook/assets/image (47).png differ diff --git a/.gitbook/assets/image (48).png b/.gitbook/assets/image (48).png new file mode 100644 index 00000000..95b9d894 Binary files /dev/null and b/.gitbook/assets/image (48).png differ diff --git a/.gitbook/assets/image (49).png b/.gitbook/assets/image (49).png new file mode 100644 index 00000000..ac491254 Binary files /dev/null and b/.gitbook/assets/image (49).png differ diff --git a/.gitbook/assets/image (5).png b/.gitbook/assets/image (5).png new file mode 100644 index 00000000..aa12316a Binary files /dev/null and b/.gitbook/assets/image (5).png differ diff --git a/.gitbook/assets/image (50).png b/.gitbook/assets/image (50).png new file mode 100644 index 00000000..1212bdb9 Binary files /dev/null and b/.gitbook/assets/image (50).png differ diff --git a/.gitbook/assets/image (51).png b/.gitbook/assets/image (51).png new file mode 100644 index 00000000..81635a39 Binary files /dev/null and b/.gitbook/assets/image (51).png differ diff --git a/.gitbook/assets/image (52).png b/.gitbook/assets/image (52).png new file mode 100644 index 00000000..018a8845 Binary files /dev/null and b/.gitbook/assets/image (52).png differ diff --git a/.gitbook/assets/image (53).png b/.gitbook/assets/image (53).png new file mode 100644 index 00000000..b5959b07 Binary files /dev/null and b/.gitbook/assets/image (53).png differ diff --git a/.gitbook/assets/image (54).png b/.gitbook/assets/image (54).png new file mode 100644 index 00000000..595a99df Binary files /dev/null and b/.gitbook/assets/image (54).png differ diff --git a/.gitbook/assets/image (55).png b/.gitbook/assets/image (55).png new file mode 100644 index 00000000..32ac8dd2 Binary files /dev/null and b/.gitbook/assets/image (55).png differ diff --git a/.gitbook/assets/image (56).png b/.gitbook/assets/image (56).png new file mode 100644 index 00000000..cc54fb78 Binary files /dev/null and b/.gitbook/assets/image (56).png differ diff --git a/.gitbook/assets/image (57).png b/.gitbook/assets/image (57).png new file mode 100644 index 00000000..e5624353 Binary files /dev/null and b/.gitbook/assets/image (57).png differ diff --git a/.gitbook/assets/image (58).png b/.gitbook/assets/image (58).png new file mode 100644 index 00000000..87379112 Binary files /dev/null and b/.gitbook/assets/image (58).png differ diff --git a/.gitbook/assets/image (59).png b/.gitbook/assets/image (59).png new file mode 100644 index 00000000..95b9d894 Binary files /dev/null and b/.gitbook/assets/image (59).png differ diff --git a/.gitbook/assets/image (6).png b/.gitbook/assets/image (6).png new file mode 100644 index 00000000..1e3263c6 Binary files /dev/null and b/.gitbook/assets/image (6).png differ diff --git a/.gitbook/assets/image (60).png b/.gitbook/assets/image (60).png new file mode 100644 index 00000000..4d2627bf Binary files /dev/null and b/.gitbook/assets/image (60).png differ diff --git a/.gitbook/assets/image (61).png b/.gitbook/assets/image (61).png new file mode 100644 index 00000000..9e1bb6f3 Binary files /dev/null and b/.gitbook/assets/image (61).png differ diff --git a/.gitbook/assets/image (62).png b/.gitbook/assets/image (62).png new file mode 100644 index 00000000..b399d7a7 Binary files /dev/null and b/.gitbook/assets/image (62).png differ diff --git a/.gitbook/assets/image (63).png b/.gitbook/assets/image (63).png new file mode 100644 index 00000000..4135c43a Binary files /dev/null and b/.gitbook/assets/image (63).png differ diff --git a/.gitbook/assets/image (64).png b/.gitbook/assets/image (64).png new file mode 100644 index 00000000..531247ae Binary files /dev/null and b/.gitbook/assets/image (64).png differ diff --git a/.gitbook/assets/image (65).png b/.gitbook/assets/image (65).png new file mode 100644 index 00000000..dcc6665c Binary files /dev/null and b/.gitbook/assets/image (65).png differ diff --git a/.gitbook/assets/image (66).png b/.gitbook/assets/image (66).png new file mode 100644 index 00000000..08a800b7 Binary files /dev/null and b/.gitbook/assets/image (66).png differ diff --git a/.gitbook/assets/image (67).png b/.gitbook/assets/image (67).png new file mode 100644 index 00000000..068ea21c Binary files /dev/null and b/.gitbook/assets/image (67).png differ diff --git a/.gitbook/assets/image (68).png b/.gitbook/assets/image (68).png new file mode 100644 index 00000000..7663a2c8 Binary files /dev/null and b/.gitbook/assets/image (68).png differ diff --git a/.gitbook/assets/image (69).png b/.gitbook/assets/image (69).png new file mode 100644 index 00000000..8ce340ad Binary files /dev/null and b/.gitbook/assets/image (69).png differ diff --git a/.gitbook/assets/image (7).png b/.gitbook/assets/image (7).png new file mode 100644 index 00000000..98ecfd36 Binary files /dev/null and b/.gitbook/assets/image (7).png differ diff --git a/.gitbook/assets/image (70).png b/.gitbook/assets/image (70).png new file mode 100644 index 00000000..06cddf77 Binary files /dev/null and b/.gitbook/assets/image (70).png differ diff --git a/.gitbook/assets/image (71).png b/.gitbook/assets/image (71).png new file mode 100644 index 00000000..5ce2e100 Binary files /dev/null and b/.gitbook/assets/image (71).png differ diff --git a/.gitbook/assets/image (72).png b/.gitbook/assets/image (72).png new file mode 100644 index 00000000..d4fe58f6 Binary files /dev/null and b/.gitbook/assets/image (72).png differ diff --git a/.gitbook/assets/image (73).png b/.gitbook/assets/image (73).png new file mode 100644 index 00000000..08956416 Binary files /dev/null and b/.gitbook/assets/image (73).png differ diff --git a/.gitbook/assets/image (74).png b/.gitbook/assets/image (74).png new file mode 100644 index 00000000..758093f8 Binary files /dev/null and b/.gitbook/assets/image (74).png differ diff --git a/.gitbook/assets/image (75).png b/.gitbook/assets/image (75).png new file mode 100644 index 00000000..758093f8 Binary files /dev/null and b/.gitbook/assets/image (75).png differ diff --git a/.gitbook/assets/image (76).png b/.gitbook/assets/image (76).png new file mode 100644 index 00000000..8b0b37b5 Binary files /dev/null and b/.gitbook/assets/image (76).png differ diff --git a/.gitbook/assets/image (77).png b/.gitbook/assets/image (77).png new file mode 100644 index 00000000..66fede8d Binary files /dev/null and b/.gitbook/assets/image (77).png differ diff --git a/.gitbook/assets/image (78).png b/.gitbook/assets/image (78).png new file mode 100644 index 00000000..ab7db93f Binary files /dev/null and b/.gitbook/assets/image (78).png differ diff --git a/.gitbook/assets/image (79).png b/.gitbook/assets/image (79).png new file mode 100644 index 00000000..7434819a Binary files /dev/null and b/.gitbook/assets/image (79).png differ diff --git a/.gitbook/assets/image (8) (1).png b/.gitbook/assets/image (8) (1).png new file mode 100644 index 00000000..1e3263c6 Binary files /dev/null and b/.gitbook/assets/image (8) (1).png differ diff --git a/.gitbook/assets/image (8).png b/.gitbook/assets/image (8).png new file mode 100644 index 00000000..ea7af571 Binary files /dev/null and b/.gitbook/assets/image (8).png differ diff --git a/.gitbook/assets/image (80).png b/.gitbook/assets/image (80).png new file mode 100644 index 00000000..7434819a Binary files /dev/null and b/.gitbook/assets/image (80).png differ diff --git a/.gitbook/assets/image (81).png b/.gitbook/assets/image (81).png new file mode 100644 index 00000000..7434819a Binary files /dev/null and b/.gitbook/assets/image (81).png differ diff --git a/.gitbook/assets/image (82).png b/.gitbook/assets/image (82).png new file mode 100644 index 00000000..7434819a Binary files /dev/null and b/.gitbook/assets/image (82).png differ diff --git a/.gitbook/assets/image (83).png b/.gitbook/assets/image (83).png new file mode 100644 index 00000000..7a2a094c Binary files /dev/null and b/.gitbook/assets/image (83).png differ diff --git a/.gitbook/assets/image (84).png b/.gitbook/assets/image (84).png new file mode 100644 index 00000000..0b65a08e Binary files /dev/null and b/.gitbook/assets/image (84).png differ diff --git a/.gitbook/assets/image (85).png b/.gitbook/assets/image (85).png new file mode 100644 index 00000000..c5e96e51 Binary files /dev/null and b/.gitbook/assets/image (85).png differ diff --git a/.gitbook/assets/image (86).png b/.gitbook/assets/image (86).png new file mode 100644 index 00000000..587f8c13 Binary files /dev/null and b/.gitbook/assets/image (86).png differ diff --git a/.gitbook/assets/image (87).png b/.gitbook/assets/image (87).png new file mode 100644 index 00000000..1acee591 Binary files /dev/null and b/.gitbook/assets/image (87).png differ diff --git a/.gitbook/assets/image (88).png b/.gitbook/assets/image (88).png new file mode 100644 index 00000000..0b240d14 Binary files /dev/null and b/.gitbook/assets/image (88).png differ diff --git a/.gitbook/assets/image (89).png b/.gitbook/assets/image (89).png new file mode 100644 index 00000000..29776689 Binary files /dev/null and b/.gitbook/assets/image (89).png differ diff --git a/.gitbook/assets/image (9).png b/.gitbook/assets/image (9).png new file mode 100644 index 00000000..cf26171b Binary files /dev/null and b/.gitbook/assets/image (9).png differ diff --git a/.gitbook/assets/image (90).png b/.gitbook/assets/image (90).png new file mode 100644 index 00000000..0e827f1b Binary files /dev/null and b/.gitbook/assets/image (90).png differ diff --git a/.gitbook/assets/image (91).png b/.gitbook/assets/image (91).png new file mode 100644 index 00000000..e8158ee0 Binary files /dev/null and b/.gitbook/assets/image (91).png differ diff --git a/.gitbook/assets/image (92).png b/.gitbook/assets/image (92).png new file mode 100644 index 00000000..8ff6db6c Binary files /dev/null and b/.gitbook/assets/image (92).png differ diff --git a/.gitbook/assets/image (93).png b/.gitbook/assets/image (93).png new file mode 100644 index 00000000..d1b9524b Binary files /dev/null and b/.gitbook/assets/image (93).png differ diff --git a/.gitbook/assets/image (94).png b/.gitbook/assets/image (94).png new file mode 100644 index 00000000..057818e0 Binary files /dev/null and b/.gitbook/assets/image (94).png differ diff --git a/.gitbook/assets/image (95).png b/.gitbook/assets/image (95).png new file mode 100644 index 00000000..0d1b5d85 Binary files /dev/null and b/.gitbook/assets/image (95).png differ diff --git a/.gitbook/assets/image (96).png b/.gitbook/assets/image (96).png new file mode 100644 index 00000000..3667f15a Binary files /dev/null and b/.gitbook/assets/image (96).png differ diff --git a/.gitbook/assets/image (97).png b/.gitbook/assets/image (97).png new file mode 100644 index 00000000..d08eadb8 Binary files /dev/null and b/.gitbook/assets/image (97).png differ diff --git a/.gitbook/assets/image (98).png b/.gitbook/assets/image (98).png new file mode 100644 index 00000000..e25336a3 Binary files /dev/null and b/.gitbook/assets/image (98).png differ diff --git a/.gitbook/assets/image (99).png b/.gitbook/assets/image (99).png new file mode 100644 index 00000000..b1e837c0 Binary files /dev/null and b/.gitbook/assets/image (99).png differ diff --git a/.gitbook/assets/image.png b/.gitbook/assets/image.png new file mode 100644 index 00000000..e8c71605 Binary files /dev/null and b/.gitbook/assets/image.png differ diff --git a/.gitbook/assets/infosec-anomaly.csv b/.gitbook/assets/infosec-anomaly.csv new file mode 100644 index 00000000..59353a16 --- /dev/null +++ b/.gitbook/assets/infosec-anomaly.csv @@ -0,0 +1,3601 @@ +Timestamp Source Workgroup Source IP Source Port Destination Workgroup Destination IP Destination Port Application Protocol Packet Size(B) +2010-12-01 23:00:00 Atlanta 10.0.0.1 58091 Texas 10.0.0.30 80 HTTP UDP 2162 +2010-12-01 23:00:01 Atlanta 10.0.0.1 65437 Texas 10.0.0.27 80 HTTP UDP 2940 +2010-12-01 23:00:02 Atlanta 10.0.0.1 52437 Texas 10.0.0.27 80 HTTP UDP 5828 +2010-12-01 23:00:03 Atlanta 10.0.0.1 58155 Texas 10.0.0.27 80 HTTP UDP 5039 +2010-12-01 23:00:04 Atlanta 10.0.0.1 51215 Texas 10.0.0.27 80 HTTP UDP 3743 +2010-12-01 23:00:05 Atlanta 10.0.0.1 58654 Texas 10.0.0.27 80 HTTP UDP 2895 +2010-12-01 23:00:06 Atlanta 10.0.0.1 61229 Texas 10.0.0.30 80 HTTP UDP 5160 +2010-12-01 23:00:07 Atlanta 10.0.0.1 61487 Texas 10.0.0.25 80 HTTP UDP 5015 +2010-12-01 23:00:08 Atlanta 10.0.0.1 56195 Texas 10.0.0.27 80 HTTP UDP 5938 +2010-12-01 23:00:09 Atlanta 10.0.0.1 61956 Texas 10.0.0.30 80 HTTP UDP 4753 +2010-12-01 23:00:10 Atlanta 10.0.0.1 56300 Texas 10.0.0.28 80 HTTP UDP 4647 +2010-12-01 23:00:11 Atlanta 10.0.0.1 58302 Texas 10.0.0.25 80 HTTP UDP 5501 +2010-12-01 23:00:12 Atlanta 10.0.0.1 55147 Texas 10.0.0.27 80 HTTP UDP 4040 +2010-12-01 23:00:13 Atlanta 10.0.0.1 52928 Texas 10.0.0.25 80 HTTP UDP 4281 +2010-12-01 23:00:14 Atlanta 10.0.0.1 54519 Texas 10.0.0.28 80 HTTP UDP 5072 +2010-12-01 23:00:15 Atlanta 10.0.0.1 53075 Texas 10.0.0.30 80 HTTP UDP 5548 +2010-12-01 23:00:16 Atlanta 10.0.0.1 60883 Texas 10.0.0.27 80 HTTP UDP 5345 +2010-12-01 23:00:17 Atlanta 10.0.0.1 57732 Texas 10.0.0.29 80 HTTP UDP 5538 +2010-12-01 23:00:18 Atlanta 10.0.0.1 65356 Texas 10.0.0.29 80 HTTP UDP 5749 +2010-12-01 23:00:19 Atlanta 10.0.0.1 50963 Texas 10.0.0.26 80 HTTP UDP 3535 +2010-12-01 23:00:20 Atlanta 10.0.0.1 61521 Texas 10.0.0.25 80 HTTP UDP 4093 +2010-12-01 23:00:21 Atlanta 10.0.0.1 58309 Texas 10.0.0.27 80 HTTP UDP 4849 +2010-12-01 23:00:22 Atlanta 10.0.0.1 59110 Texas 10.0.0.27 80 HTTP UDP 4042 +2010-12-01 23:00:23 Atlanta 10.0.0.1 65152 Texas 10.0.0.25 80 HTTP UDP 3729 +2010-12-01 23:00:24 Atlanta 10.0.0.1 52299 Texas 10.0.0.25 80 HTTP UDP 3419 +2010-12-01 23:00:25 Atlanta 10.0.0.1 53020 Texas 10.0.0.27 80 HTTP UDP 2708 +2010-12-01 23:00:26 Atlanta 10.0.0.1 51644 Texas 10.0.0.30 80 HTTP UDP 3183 +2010-12-01 23:00:27 Atlanta 10.0.0.1 60407 Texas 10.0.0.27 80 HTTP UDP 3326 +2010-12-01 23:00:28 Atlanta 10.0.0.1 60449 Texas 10.0.0.25 80 HTTP UDP 4189 +2010-12-01 23:00:29 Atlanta 10.0.0.1 56257 Texas 10.0.0.27 80 HTTP UDP 4289 +2010-12-01 23:00:30 Atlanta 10.0.0.1 57596 Texas 10.0.0.30 80 HTTP UDP 5906 +2010-12-01 23:00:31 Atlanta 10.0.0.1 63953 Texas 10.0.0.28 80 HTTP UDP 2295 +2010-12-01 23:00:32 Atlanta 10.0.0.1 62129 Texas 10.0.0.25 80 HTTP UDP 2790 +2010-12-01 23:00:33 Atlanta 10.0.0.1 62075 Texas 10.0.0.25 80 HTTP UDP 5951 +2010-12-01 23:00:34 Atlanta 10.0.0.1 52314 Texas 10.0.0.30 80 HTTP UDP 3374 +2010-12-01 23:00:35 Atlanta 10.0.0.1 52687 Texas 10.0.0.25 80 HTTP UDP 3146 +2010-12-01 23:00:36 Atlanta 10.0.0.1 56812 Texas 10.0.0.28 80 HTTP UDP 4393 +2010-12-01 23:00:37 Atlanta 10.0.0.1 50252 Texas 10.0.0.30 80 HTTP UDP 5482 +2010-12-01 23:00:38 Atlanta 10.0.0.1 53399 Texas 10.0.0.27 80 HTTP UDP 4424 +2010-12-01 23:00:39 Atlanta 10.0.0.1 61979 Texas 10.0.0.25 80 HTTP UDP 5841 +2010-12-01 23:00:40 Atlanta 10.0.0.1 62790 Texas 10.0.0.25 80 HTTP UDP 3019 +2010-12-01 23:00:41 Atlanta 10.0.0.1 59326 Texas 10.0.0.26 80 HTTP UDP 3899 +2010-12-01 23:00:42 Atlanta 10.0.0.1 55736 Texas 10.0.0.28 80 HTTP UDP 5132 +2010-12-01 23:00:43 Atlanta 10.0.0.1 63651 Texas 10.0.0.28 80 HTTP UDP 5501 +2010-12-01 23:00:44 Atlanta 10.0.0.1 55889 Texas 10.0.0.25 80 HTTP UDP 3540 +2010-12-01 23:00:45 Atlanta 10.0.0.1 56207 Texas 10.0.0.30 80 HTTP UDP 2895 +2010-12-01 23:00:46 Atlanta 10.0.0.1 58187 Texas 10.0.0.27 80 HTTP UDP 2263 +2010-12-01 23:00:47 Atlanta 10.0.0.1 59555 Texas 10.0.0.28 80 HTTP UDP 5617 +2010-12-01 23:00:48 Atlanta 10.0.0.1 62241 Texas 10.0.0.27 80 HTTP UDP 4855 +2010-12-01 23:00:49 Atlanta 10.0.0.1 60006 Texas 10.0.0.28 80 HTTP UDP 2477 +2010-12-01 23:00:50 Atlanta 10.0.0.1 50953 Texas 10.0.0.27 80 HTTP UDP 2172 +2010-12-01 23:00:51 Atlanta 10.0.0.1 58403 Texas 10.0.0.26 80 HTTP UDP 5806 +2010-12-01 23:00:52 Atlanta 10.0.0.1 51377 Texas 10.0.0.27 80 HTTP UDP 5853 +2010-12-01 23:00:53 Atlanta 10.0.0.1 49713 Texas 10.0.0.27 80 HTTP UDP 2240 +2010-12-01 23:00:54 Atlanta 10.0.0.1 50339 Texas 10.0.0.27 80 HTTP UDP 3523 +2010-12-01 23:00:55 Atlanta 10.0.0.1 51184 Texas 10.0.0.30 80 HTTP UDP 4530 +2010-12-01 23:00:56 Atlanta 10.0.0.1 64381 Texas 10.0.0.30 80 HTTP UDP 5106 +2010-12-01 23:00:57 Atlanta 10.0.0.1 50289 Texas 10.0.0.26 80 HTTP UDP 2830 +2010-12-01 23:00:58 Atlanta 10.0.0.1 64092 Texas 10.0.0.29 80 HTTP UDP 3896 +2010-12-01 23:00:59 Atlanta 10.0.0.1 60986 Texas 10.0.0.27 80 HTTP UDP 4564 +2010-12-01 23:01:00 Atlanta 10.0.0.1 65267 Texas 10.0.0.30 80 HTTP UDP 5394 +2010-12-01 23:01:01 Atlanta 10.0.0.1 53499 Texas 10.0.0.27 80 HTTP UDP 5807 +2010-12-01 23:01:02 Atlanta 10.0.0.1 53005 Texas 10.0.0.26 80 HTTP UDP 3469 +2010-12-01 23:01:03 Atlanta 10.0.0.1 64903 Texas 10.0.0.25 80 HTTP UDP 2701 +2010-12-01 23:01:04 Atlanta 10.0.0.1 64077 Texas 10.0.0.25 80 HTTP UDP 5556 +2010-12-01 23:01:05 Atlanta 10.0.0.1 63161 Texas 10.0.0.26 80 HTTP UDP 3809 +2010-12-01 23:01:06 Atlanta 10.0.0.1 50000 Texas 10.0.0.29 80 HTTP UDP 3901 +2010-12-01 23:01:07 Atlanta 10.0.0.1 64447 Texas 10.0.0.28 80 HTTP UDP 4893 +2010-12-01 23:01:08 Atlanta 10.0.0.1 60231 Texas 10.0.0.29 80 HTTP UDP 5346 +2010-12-01 23:01:09 Atlanta 10.0.0.1 50271 Texas 10.0.0.27 80 HTTP UDP 5422 +2010-12-01 23:01:10 Atlanta 10.0.0.1 57146 Texas 10.0.0.28 80 HTTP UDP 3253 +2010-12-01 23:01:11 Atlanta 10.0.0.1 62234 Texas 10.0.0.25 80 HTTP UDP 3121 +2010-12-01 23:01:12 Atlanta 10.0.0.1 60055 Texas 10.0.0.25 80 HTTP UDP 5841 +2010-12-01 23:01:13 Atlanta 10.0.0.1 62314 Texas 10.0.0.28 80 HTTP UDP 4651 +2010-12-01 23:01:14 Atlanta 10.0.0.1 50400 Texas 10.0.0.29 80 HTTP UDP 5480 +2010-12-01 23:01:15 Atlanta 10.0.0.1 58678 Texas 10.0.0.26 80 HTTP UDP 2215 +2010-12-01 23:01:16 Atlanta 10.0.0.1 65454 Texas 10.0.0.30 80 HTTP UDP 5712 +2010-12-01 23:01:17 Atlanta 10.0.0.1 64610 Texas 10.0.0.29 80 HTTP UDP 4827 +2010-12-01 23:01:18 Atlanta 10.0.0.1 63058 Texas 10.0.0.29 80 HTTP UDP 2421 +2010-12-01 23:01:19 Atlanta 10.0.0.1 58644 Texas 10.0.0.30 80 HTTP UDP 4696 +2010-12-01 23:01:20 Atlanta 10.0.0.1 53560 Texas 10.0.0.29 80 HTTP UDP 5197 +2010-12-01 23:01:21 Atlanta 10.0.0.1 49962 Texas 10.0.0.28 80 HTTP UDP 4515 +2010-12-01 23:01:22 Atlanta 10.0.0.1 56778 Texas 10.0.0.25 80 HTTP UDP 4251 +2010-12-01 23:01:23 Atlanta 10.0.0.1 63950 Texas 10.0.0.30 80 HTTP UDP 5290 +2010-12-01 23:01:24 Atlanta 10.0.0.1 64406 Texas 10.0.0.26 80 HTTP UDP 3010 +2010-12-01 23:01:25 Atlanta 10.0.0.1 54031 Texas 10.0.0.29 80 HTTP UDP 3013 +2010-12-01 23:01:26 Atlanta 10.0.0.1 57871 Texas 10.0.0.29 80 HTTP UDP 4886 +2010-12-01 23:01:27 Atlanta 10.0.0.1 64184 Texas 10.0.0.29 80 HTTP UDP 3486 +2010-12-01 23:01:28 Atlanta 10.0.0.1 51807 Texas 10.0.0.30 80 HTTP UDP 5327 +2010-12-01 23:01:29 Atlanta 10.0.0.1 50496 Texas 10.0.0.28 80 HTTP UDP 2877 +2010-12-01 23:01:30 Atlanta 10.0.0.1 60938 Texas 10.0.0.25 80 HTTP UDP 4097 +2010-12-01 23:01:31 Atlanta 10.0.0.1 64331 Texas 10.0.0.30 80 HTTP UDP 3268 +2010-12-01 23:01:32 Atlanta 10.0.0.1 55162 Texas 10.0.0.30 80 HTTP UDP 4886 +2010-12-01 23:01:33 Atlanta 10.0.0.1 60274 Texas 10.0.0.25 80 HTTP UDP 5935 +2010-12-01 23:01:34 Atlanta 10.0.0.1 52038 Texas 10.0.0.27 80 HTTP UDP 2956 +2010-12-01 23:01:35 Atlanta 10.0.0.1 59052 Texas 10.0.0.28 80 HTTP UDP 2924 +2010-12-01 23:01:36 Atlanta 10.0.0.1 57104 Texas 10.0.0.27 80 HTTP UDP 2881 +2010-12-01 23:01:37 Atlanta 10.0.0.1 52978 Texas 10.0.0.29 80 HTTP UDP 5284 +2010-12-01 23:01:38 Atlanta 10.0.0.1 56424 Texas 10.0.0.27 80 HTTP UDP 2322 +2010-12-01 23:01:39 Atlanta 10.0.0.1 59631 Texas 10.0.0.25 80 HTTP UDP 5628 +2010-12-01 23:01:40 Atlanta 10.0.0.1 49471 Texas 10.0.0.28 80 HTTP UDP 2956 +2010-12-01 23:01:41 Atlanta 10.0.0.1 61732 Texas 10.0.0.26 80 HTTP UDP 2652 +2010-12-01 23:01:42 Atlanta 10.0.0.1 57781 Texas 10.0.0.29 80 HTTP UDP 2038 +2010-12-01 23:01:43 Atlanta 10.0.0.1 54441 Texas 10.0.0.27 80 HTTP UDP 4771 +2010-12-01 23:01:44 Atlanta 10.0.0.1 52907 Texas 10.0.0.28 80 HTTP UDP 5540 +2010-12-01 23:01:45 Atlanta 10.0.0.1 61499 Texas 10.0.0.29 80 HTTP UDP 4287 +2010-12-01 23:01:46 Atlanta 10.0.0.1 54134 Texas 10.0.0.27 80 HTTP UDP 3568 +2010-12-01 23:01:47 Atlanta 10.0.0.1 62705 Texas 10.0.0.30 80 HTTP UDP 4598 +2010-12-01 23:01:48 Atlanta 10.0.0.1 51139 Texas 10.0.0.30 80 HTTP UDP 3864 +2010-12-01 23:01:49 Atlanta 10.0.0.1 61524 Texas 10.0.0.30 80 HTTP UDP 3171 +2010-12-01 23:01:50 Atlanta 10.0.0.1 57084 Texas 10.0.0.28 80 HTTP UDP 2934 +2010-12-01 23:01:51 Atlanta 10.0.0.1 65037 Texas 10.0.0.30 80 HTTP UDP 3970 +2010-12-01 23:01:52 Atlanta 10.0.0.1 63733 Texas 10.0.0.25 80 HTTP UDP 4774 +2010-12-01 23:01:53 Atlanta 10.0.0.1 63231 Texas 10.0.0.28 80 HTTP UDP 2001 +2010-12-01 23:01:54 Atlanta 10.0.0.1 62638 Texas 10.0.0.28 80 HTTP UDP 5841 +2010-12-01 23:01:55 Atlanta 10.0.0.1 49186 Texas 10.0.0.25 80 HTTP UDP 2554 +2010-12-01 23:01:56 Atlanta 10.0.0.1 59049 Texas 10.0.0.30 80 HTTP UDP 2975 +2010-12-01 23:01:57 Atlanta 10.0.0.1 52509 Texas 10.0.0.25 80 HTTP UDP 3415 +2010-12-01 23:01:58 Atlanta 10.0.0.1 53384 Texas 10.0.0.26 80 HTTP UDP 4948 +2010-12-01 23:01:59 Atlanta 10.0.0.1 55364 Texas 10.0.0.28 80 HTTP UDP 4893 +2010-12-01 23:02:00 Atlanta 10.0.0.1 59979 Texas 10.0.0.28 80 HTTP UDP 5543 +2010-12-01 23:02:01 Atlanta 10.0.0.1 55244 Texas 10.0.0.27 80 HTTP UDP 4903 +2010-12-01 23:02:02 Atlanta 10.0.0.1 65421 Texas 10.0.0.27 80 HTTP UDP 2602 +2010-12-01 23:02:03 Atlanta 10.0.0.1 62404 Texas 10.0.0.26 80 HTTP UDP 4150 +2010-12-01 23:02:04 Atlanta 10.0.0.1 56126 Texas 10.0.0.27 80 HTTP UDP 4415 +2010-12-01 23:02:05 Atlanta 10.0.0.1 56581 Texas 10.0.0.28 80 HTTP UDP 2562 +2010-12-01 23:02:06 Atlanta 10.0.0.1 51095 Texas 10.0.0.29 80 HTTP UDP 4379 +2010-12-01 23:02:07 Atlanta 10.0.0.1 63279 Texas 10.0.0.27 80 HTTP UDP 5223 +2010-12-01 23:02:08 Atlanta 10.0.0.1 58623 Texas 10.0.0.29 80 HTTP UDP 3398 +2010-12-01 23:02:09 Atlanta 10.0.0.1 49183 Texas 10.0.0.29 80 HTTP UDP 3171 +2010-12-01 23:02:10 Atlanta 10.0.0.1 65403 Texas 10.0.0.30 80 HTTP UDP 5483 +2010-12-01 23:02:11 Atlanta 10.0.0.1 59350 Texas 10.0.0.28 80 HTTP UDP 3634 +2010-12-01 23:02:12 Atlanta 10.0.0.1 64758 Texas 10.0.0.27 80 HTTP UDP 5594 +2010-12-01 23:02:13 Atlanta 10.0.0.1 52317 Texas 10.0.0.29 80 HTTP UDP 5634 +2010-12-01 23:02:14 Atlanta 10.0.0.1 51617 Texas 10.0.0.29 80 HTTP UDP 2782 +2010-12-01 23:02:15 Atlanta 10.0.0.1 52223 Texas 10.0.0.28 80 HTTP UDP 4032 +2010-12-01 23:02:16 Atlanta 10.0.0.1 53077 Texas 10.0.0.28 80 HTTP UDP 4399 +2010-12-01 23:02:17 Atlanta 10.0.0.1 57679 Texas 10.0.0.27 80 HTTP UDP 4328 +2010-12-01 23:02:18 Atlanta 10.0.0.1 59238 Texas 10.0.0.28 80 HTTP UDP 5096 +2010-12-01 23:02:19 Atlanta 10.0.0.1 61148 Texas 10.0.0.25 80 HTTP UDP 2186 +2010-12-01 23:02:20 Atlanta 10.0.0.1 49776 Texas 10.0.0.25 80 HTTP UDP 3764 +2010-12-01 23:02:21 Atlanta 10.0.0.1 61343 Texas 10.0.0.27 80 HTTP UDP 4074 +2010-12-01 23:02:22 Atlanta 10.0.0.1 57767 Texas 10.0.0.29 80 HTTP UDP 3306 +2010-12-01 23:02:23 Atlanta 10.0.0.1 51208 Texas 10.0.0.30 80 HTTP UDP 3189 +2010-12-01 23:02:24 Atlanta 10.0.0.1 63594 Texas 10.0.0.26 80 HTTP UDP 2361 +2010-12-01 23:02:25 Atlanta 10.0.0.1 63501 Texas 10.0.0.30 80 HTTP UDP 4379 +2010-12-01 23:02:26 Atlanta 10.0.0.1 59718 Texas 10.0.0.28 80 HTTP UDP 3230 +2010-12-01 23:02:27 Atlanta 10.0.0.1 59262 Texas 10.0.0.26 80 HTTP UDP 5603 +2010-12-01 23:02:28 Atlanta 10.0.0.1 49954 Texas 10.0.0.27 80 HTTP UDP 4899 +2010-12-01 23:02:29 Atlanta 10.0.0.1 61248 Texas 10.0.0.28 80 HTTP UDP 2656 +2010-12-01 23:02:30 Atlanta 10.0.0.1 64059 Texas 10.0.0.29 80 HTTP UDP 2806 +2010-12-01 23:02:31 Atlanta 10.0.0.1 53370 Texas 10.0.0.30 80 HTTP UDP 5326 +2010-12-01 23:02:32 Atlanta 10.0.0.1 62138 Texas 10.0.0.29 80 HTTP UDP 5761 +2010-12-01 23:02:33 Atlanta 10.0.0.1 56588 Texas 10.0.0.30 80 HTTP UDP 4173 +2010-12-01 23:02:34 Atlanta 10.0.0.1 63841 Texas 10.0.0.29 80 HTTP UDP 3326 +2010-12-01 23:02:35 Atlanta 10.0.0.1 50916 Texas 10.0.0.29 80 HTTP UDP 5556 +2010-12-01 23:02:36 Atlanta 10.0.0.1 56089 Texas 10.0.0.25 80 HTTP UDP 4488 +2010-12-01 23:02:37 Atlanta 10.0.0.1 56777 Texas 10.0.0.26 80 HTTP UDP 2778 +2010-12-01 23:02:38 Atlanta 10.0.0.1 57543 Texas 10.0.0.30 80 HTTP UDP 4826 +2010-12-01 23:02:39 Atlanta 10.0.0.1 58068 Texas 10.0.0.26 80 HTTP UDP 5850 +2010-12-01 23:02:40 Atlanta 10.0.0.1 55887 Texas 10.0.0.27 80 HTTP UDP 5667 +2010-12-01 23:02:41 Atlanta 10.0.0.1 50231 Texas 10.0.0.29 80 HTTP UDP 2046 +2010-12-01 23:02:42 Atlanta 10.0.0.1 51590 Texas 10.0.0.26 80 HTTP UDP 2909 +2010-12-01 23:02:43 Atlanta 10.0.0.1 54491 Texas 10.0.0.25 80 HTTP UDP 5584 +2010-12-01 23:02:44 Atlanta 10.0.0.1 59684 Texas 10.0.0.28 80 HTTP UDP 5258 +2010-12-01 23:02:45 Atlanta 10.0.0.1 55515 Texas 10.0.0.29 80 HTTP UDP 4635 +2010-12-01 23:02:46 Atlanta 10.0.0.1 53836 Texas 10.0.0.30 80 HTTP UDP 2940 +2010-12-01 23:02:47 Atlanta 10.0.0.1 55650 Texas 10.0.0.27 80 HTTP UDP 3153 +2010-12-01 23:02:48 Atlanta 10.0.0.1 60357 Texas 10.0.0.28 80 HTTP UDP 4032 +2010-12-01 23:02:49 Atlanta 10.0.0.1 63156 Texas 10.0.0.27 80 HTTP UDP 3855 +2010-12-01 23:02:50 Atlanta 10.0.0.1 57289 Texas 10.0.0.29 80 HTTP UDP 5197 +2010-12-01 23:02:51 Atlanta 10.0.0.1 60318 Texas 10.0.0.30 80 HTTP UDP 3293 +2010-12-01 23:02:52 Atlanta 10.0.0.1 61101 Texas 10.0.0.30 80 HTTP UDP 4724 +2010-12-01 23:02:53 Atlanta 10.0.0.1 55402 Texas 10.0.0.29 80 HTTP UDP 2129 +2010-12-01 23:02:54 Atlanta 10.0.0.1 58632 Texas 10.0.0.26 80 HTTP UDP 4801 +2010-12-01 23:02:55 Atlanta 10.0.0.1 62314 Texas 10.0.0.30 80 HTTP UDP 3331 +2010-12-01 23:02:56 Atlanta 10.0.0.1 62887 Texas 10.0.0.30 80 HTTP UDP 5673 +2010-12-01 23:02:57 Atlanta 10.0.0.1 55596 Texas 10.0.0.30 80 HTTP UDP 2693 +2010-12-01 23:02:58 Atlanta 10.0.0.1 52934 Texas 10.0.0.25 80 HTTP UDP 4090 +2010-12-01 23:02:59 Atlanta 10.0.0.1 58650 Texas 10.0.0.29 80 HTTP UDP 3142 +2010-12-01 23:03:00 Atlanta 10.0.0.1 50097 Texas 10.0.0.28 80 HTTP UDP 3872 +2010-12-01 23:03:01 Atlanta 10.0.0.1 51347 Texas 10.0.0.26 80 HTTP UDP 4920 +2010-12-01 23:03:02 Atlanta 10.0.0.1 63236 Texas 10.0.0.29 80 HTTP UDP 3398 +2010-12-01 23:03:03 Atlanta 10.0.0.1 53304 Texas 10.0.0.29 80 HTTP UDP 4651 +2010-12-01 23:03:04 Atlanta 10.0.0.1 55690 Texas 10.0.0.26 80 HTTP UDP 3855 +2010-12-01 23:03:05 Atlanta 10.0.0.1 55372 Texas 10.0.0.27 80 HTTP UDP 2498 +2010-12-01 23:03:06 Atlanta 10.0.0.1 56815 Texas 10.0.0.28 80 HTTP UDP 3182 +2010-12-01 23:03:07 Atlanta 10.0.0.1 60557 Texas 10.0.0.29 80 HTTP UDP 3743 +2010-12-01 23:03:08 Atlanta 10.0.0.1 64842 Texas 10.0.0.25 80 HTTP UDP 2355 +2010-12-01 23:03:09 Atlanta 10.0.0.1 56483 Texas 10.0.0.25 80 HTTP UDP 2579 +2010-12-01 23:03:10 Atlanta 10.0.0.1 52945 Texas 10.0.0.26 80 HTTP UDP 4496 +2010-12-01 23:03:11 Atlanta 10.0.0.1 63727 Texas 10.0.0.30 80 HTTP UDP 2882 +2010-12-01 23:03:12 Atlanta 10.0.0.1 54484 Texas 10.0.0.28 80 HTTP UDP 3814 +2010-12-01 23:03:13 Atlanta 10.0.0.1 52835 Texas 10.0.0.26 80 HTTP UDP 2326 +2010-12-01 23:03:14 Atlanta 10.0.0.1 55572 Texas 10.0.0.28 80 HTTP UDP 2432 +2010-12-01 23:03:15 Atlanta 10.0.0.1 59535 Texas 10.0.0.25 80 HTTP UDP 5321 +2010-12-01 23:03:16 Atlanta 10.0.0.1 60150 Texas 10.0.0.25 80 HTTP UDP 4936 +2010-12-01 23:03:17 Atlanta 10.0.0.1 50522 Texas 10.0.0.27 80 HTTP UDP 5747 +2010-12-01 23:03:18 Atlanta 10.0.0.1 55069 Texas 10.0.0.25 80 HTTP UDP 3696 +2010-12-01 23:03:19 Atlanta 10.0.0.1 50079 Texas 10.0.0.29 80 HTTP UDP 4369 +2010-12-01 23:03:20 Atlanta 10.0.0.1 62790 Texas 10.0.0.28 80 HTTP UDP 3100 +2010-12-01 23:03:21 Atlanta 10.0.0.1 52844 Texas 10.0.0.30 80 HTTP UDP 3696 +2010-12-01 23:03:22 Atlanta 10.0.0.1 51410 Texas 10.0.0.26 80 HTTP UDP 5558 +2010-12-01 23:03:23 Atlanta 10.0.0.1 50005 Texas 10.0.0.28 80 HTTP UDP 5082 +2010-12-01 23:03:24 Atlanta 10.0.0.1 61534 Texas 10.0.0.28 80 HTTP UDP 2778 +2010-12-01 23:03:25 Atlanta 10.0.0.1 51593 Texas 10.0.0.25 80 HTTP UDP 3156 +2010-12-01 23:03:26 Atlanta 10.0.0.1 63180 Texas 10.0.0.28 80 HTTP UDP 5759 +2010-12-01 23:03:27 Atlanta 10.0.0.1 64477 Texas 10.0.0.30 80 HTTP UDP 3415 +2010-12-01 23:03:28 Atlanta 10.0.0.1 49418 Texas 10.0.0.25 80 HTTP UDP 5460 +2010-12-01 23:03:29 Atlanta 10.0.0.1 55998 Texas 10.0.0.26 80 HTTP UDP 4077 +2010-12-01 23:03:30 Atlanta 10.0.0.1 57793 Texas 10.0.0.25 80 HTTP UDP 3983 +2010-12-01 23:03:31 Atlanta 10.0.0.1 55443 Texas 10.0.0.27 80 HTTP UDP 3921 +2010-12-01 23:03:32 Atlanta 10.0.0.1 64484 Texas 10.0.0.29 80 HTTP UDP 2956 +2010-12-01 23:03:33 Atlanta 10.0.0.1 51368 Texas 10.0.0.27 80 HTTP UDP 5837 +2010-12-01 23:03:34 Atlanta 10.0.0.1 61405 Texas 10.0.0.28 80 HTTP UDP 5252 +2010-12-01 23:03:35 Atlanta 10.0.0.1 61574 Texas 10.0.0.28 80 HTTP UDP 4717 +2010-12-01 23:03:36 Atlanta 10.0.0.1 60732 Texas 10.0.0.30 80 HTTP UDP 3139 +2010-12-01 23:03:37 Atlanta 10.0.0.1 51816 Texas 10.0.0.28 80 HTTP UDP 5171 +2010-12-01 23:03:38 Atlanta 10.0.0.1 62329 Texas 10.0.0.29 80 HTTP UDP 3131 +2010-12-01 23:03:39 Atlanta 10.0.0.1 56557 Texas 10.0.0.27 80 HTTP UDP 5710 +2010-12-01 23:03:40 Atlanta 10.0.0.1 58879 Texas 10.0.0.26 80 HTTP UDP 3901 +2010-12-01 23:03:41 Atlanta 10.0.0.1 50856 Texas 10.0.0.27 80 HTTP UDP 5552 +2010-12-01 23:03:42 Atlanta 10.0.0.1 60626 Texas 10.0.0.25 80 HTTP UDP 3781 +2010-12-01 23:03:43 Atlanta 10.0.0.1 52548 Texas 10.0.0.28 80 HTTP UDP 2532 +2010-12-01 23:03:44 Atlanta 10.0.0.1 60804 Texas 10.0.0.27 80 HTTP UDP 2603 +2010-12-01 23:03:45 Atlanta 10.0.0.1 53518 Texas 10.0.0.25 80 HTTP UDP 3317 +2010-12-01 23:03:46 Atlanta 10.0.0.1 60155 Texas 10.0.0.29 80 HTTP UDP 2257 +2010-12-01 23:03:47 Atlanta 10.0.0.1 55573 Texas 10.0.0.25 80 HTTP UDP 5341 +2010-12-01 23:03:48 Atlanta 10.0.0.1 55252 Texas 10.0.0.26 80 HTTP UDP 5460 +2010-12-01 23:03:49 Atlanta 10.0.0.1 55375 Texas 10.0.0.30 80 HTTP UDP 4933 +2010-12-01 23:03:50 Atlanta 10.0.0.1 63554 Texas 10.0.0.28 80 HTTP UDP 3657 +2010-12-01 23:03:51 Atlanta 10.0.0.1 52757 Texas 10.0.0.28 80 HTTP UDP 3940 +2010-12-01 23:03:52 Atlanta 10.0.0.1 57002 Texas 10.0.0.25 80 HTTP UDP 4934 +2010-12-01 23:03:53 Atlanta 10.0.0.1 53127 Texas 10.0.0.25 80 HTTP UDP 3357 +2010-12-01 23:03:54 Atlanta 10.0.0.1 55984 Texas 10.0.0.26 80 HTTP UDP 3476 +2010-12-01 23:03:55 Atlanta 10.0.0.1 54393 Texas 10.0.0.26 80 HTTP UDP 5047 +2010-12-01 23:03:56 Atlanta 10.0.0.1 52513 Texas 10.0.0.28 80 HTTP UDP 4088 +2010-12-01 23:03:57 Atlanta 10.0.0.1 56114 Texas 10.0.0.26 80 HTTP UDP 2940 +2010-12-01 23:03:58 Atlanta 10.0.0.1 51979 Texas 10.0.0.27 80 HTTP UDP 4657 +2010-12-01 23:03:59 Atlanta 10.0.0.1 51445 Texas 10.0.0.29 80 HTTP UDP 3955 +2010-12-01 23:04:00 Atlanta 10.0.0.1 53987 Texas 10.0.0.28 80 HTTP UDP 5811 +2010-12-01 23:04:01 Atlanta 10.0.0.1 63380 Texas 10.0.0.29 80 HTTP UDP 5435 +2010-12-01 23:04:02 Atlanta 10.0.0.1 63224 Texas 10.0.0.29 80 HTTP UDP 4381 +2010-12-01 23:04:03 Atlanta 10.0.0.1 56686 Texas 10.0.0.30 80 HTTP UDP 3147 +2010-12-01 23:04:04 Atlanta 10.0.0.1 64799 Texas 10.0.0.30 80 HTTP UDP 3959 +2010-12-01 23:04:05 Atlanta 10.0.0.1 55116 Texas 10.0.0.26 80 HTTP UDP 3947 +2010-12-01 23:04:06 Atlanta 10.0.0.1 59761 Texas 10.0.0.26 80 HTTP UDP 3509 +2010-12-01 23:04:07 Atlanta 10.0.0.1 53135 Texas 10.0.0.28 80 HTTP UDP 5888 +2010-12-01 23:04:08 Atlanta 10.0.0.1 60928 Texas 10.0.0.27 80 HTTP UDP 5660 +2010-12-01 23:04:09 Atlanta 10.0.0.1 65100 Texas 10.0.0.26 80 HTTP UDP 2882 +2010-12-01 23:04:10 Atlanta 10.0.0.1 62869 Texas 10.0.0.25 80 HTTP UDP 4936 +2010-12-01 23:04:11 Atlanta 10.0.0.1 49434 Texas 10.0.0.29 80 HTTP UDP 5347 +2010-12-01 23:04:12 Atlanta 10.0.0.1 56999 Texas 10.0.0.29 80 HTTP UDP 3723 +2010-12-01 23:04:13 Atlanta 10.0.0.1 61679 Texas 10.0.0.25 80 HTTP UDP 2587 +2010-12-01 23:04:14 Atlanta 10.0.0.1 50851 Texas 10.0.0.29 80 HTTP UDP 2895 +2010-12-01 23:04:15 Atlanta 10.0.0.1 59733 Texas 10.0.0.25 80 HTTP UDP 3523 +2010-12-01 23:04:16 Atlanta 10.0.0.1 61639 Texas 10.0.0.27 80 HTTP UDP 4524 +2010-12-01 23:04:17 Atlanta 10.0.0.1 50971 Texas 10.0.0.26 80 HTTP UDP 2561 +2010-12-01 23:04:18 Atlanta 10.0.0.1 50454 Texas 10.0.0.28 80 HTTP UDP 2015 +2010-12-01 23:04:19 Atlanta 10.0.0.1 49390 Texas 10.0.0.25 80 HTTP UDP 4043 +2010-12-01 23:04:20 Atlanta 10.0.0.1 62590 Texas 10.0.0.25 80 HTTP UDP 3144 +2010-12-01 23:04:21 Atlanta 10.0.0.1 52263 Texas 10.0.0.28 80 HTTP UDP 5029 +2010-12-01 23:04:22 Atlanta 10.0.0.1 62724 Texas 10.0.0.30 80 HTTP UDP 2690 +2010-12-01 23:04:23 Atlanta 10.0.0.1 59047 Texas 10.0.0.26 80 HTTP UDP 5559 +2010-12-01 23:04:24 Atlanta 10.0.0.1 63703 Texas 10.0.0.28 80 HTTP UDP 3292 +2010-12-01 23:04:25 Atlanta 10.0.0.1 50370 Texas 10.0.0.29 80 HTTP UDP 3861 +2010-12-01 23:04:26 Atlanta 10.0.0.1 65264 Texas 10.0.0.26 80 HTTP UDP 2409 +2010-12-01 23:04:27 Atlanta 10.0.0.1 59055 Texas 10.0.0.26 80 HTTP UDP 5738 +2010-12-01 23:04:28 Atlanta 10.0.0.1 57150 Texas 10.0.0.28 80 HTTP UDP 4920 +2010-12-01 23:04:29 Atlanta 10.0.0.1 61238 Texas 10.0.0.29 80 HTTP UDP 5128 +2010-12-01 23:04:30 Atlanta 10.0.0.1 54255 Texas 10.0.0.27 80 HTTP UDP 2576 +2010-12-01 23:04:31 Atlanta 10.0.0.1 61505 Texas 10.0.0.29 80 HTTP UDP 4902 +2010-12-01 23:04:32 Atlanta 10.0.0.1 58761 Texas 10.0.0.25 80 HTTP UDP 2660 +2010-12-01 23:04:33 Atlanta 10.0.0.1 53290 Texas 10.0.0.30 80 HTTP UDP 5243 +2010-12-01 23:04:34 Atlanta 10.0.0.1 58129 Texas 10.0.0.25 80 HTTP UDP 5422 +2010-12-01 23:04:35 Atlanta 10.0.0.1 58260 Texas 10.0.0.28 80 HTTP UDP 2784 +2010-12-01 23:04:36 Atlanta 10.0.0.1 49459 Texas 10.0.0.25 80 HTTP UDP 4327 +2010-12-01 23:04:37 Atlanta 10.0.0.1 54885 Texas 10.0.0.30 80 HTTP UDP 4013 +2010-12-01 23:04:38 Atlanta 10.0.0.1 63137 Texas 10.0.0.28 80 HTTP UDP 4922 +2010-12-01 23:04:39 Atlanta 10.0.0.1 53037 Texas 10.0.0.27 80 HTTP UDP 5273 +2010-12-01 23:04:40 Atlanta 10.0.0.1 61341 Texas 10.0.0.25 80 HTTP UDP 2664 +2010-12-01 23:04:41 Atlanta 10.0.0.1 65021 Texas 10.0.0.26 80 HTTP UDP 5632 +2010-12-01 23:04:42 Atlanta 10.0.0.1 50521 Texas 10.0.0.26 80 HTTP UDP 3568 +2010-12-01 23:04:43 Atlanta 10.0.0.1 59530 Texas 10.0.0.30 80 HTTP UDP 5556 +2010-12-01 23:04:44 Atlanta 10.0.0.1 53111 Texas 10.0.0.26 80 HTTP UDP 3658 +2010-12-01 23:04:45 Atlanta 10.0.0.1 58171 Texas 10.0.0.25 80 HTTP UDP 4238 +2010-12-01 23:04:46 Atlanta 10.0.0.1 53233 Texas 10.0.0.25 80 HTTP UDP 4934 +2010-12-01 23:04:47 Atlanta 10.0.0.1 63719 Texas 10.0.0.27 80 HTTP UDP 5511 +2010-12-01 23:04:48 Atlanta 10.0.0.1 65453 Texas 10.0.0.29 80 HTTP UDP 5888 +2010-12-01 23:04:49 Atlanta 10.0.0.1 62463 Texas 10.0.0.26 80 HTTP UDP 4363 +2010-12-01 23:04:50 Atlanta 10.0.0.1 64242 Texas 10.0.0.28 80 HTTP UDP 4287 +2010-12-01 23:04:51 Atlanta 10.0.0.1 55297 Texas 10.0.0.28 80 HTTP UDP 3814 +2010-12-01 23:04:52 Atlanta 10.0.0.1 59470 Texas 10.0.0.26 80 HTTP UDP 5556 +2010-12-01 23:04:53 Atlanta 10.0.0.1 57796 Texas 10.0.0.26 80 HTTP UDP 2906 +2010-12-01 23:04:54 Atlanta 10.0.0.1 49230 Texas 10.0.0.25 80 HTTP UDP 4936 +2010-12-01 23:04:55 Atlanta 10.0.0.1 62201 Texas 10.0.0.26 80 HTTP UDP 5810 +2010-12-01 23:04:56 Atlanta 10.0.0.1 63360 Texas 10.0.0.27 80 HTTP UDP 2366 +2010-12-01 23:04:57 Atlanta 10.0.0.1 55995 Texas 10.0.0.28 80 HTTP UDP 3066 +2010-12-01 23:04:58 Atlanta 10.0.0.1 65089 Texas 10.0.0.29 80 HTTP UDP 3673 +2010-12-01 23:04:59 Atlanta 10.0.0.1 56664 Texas 10.0.0.27 80 HTTP UDP 2782 +2010-12-01 23:05:00 Atlanta 10.0.0.1 59262 Texas 10.0.0.29 80 HTTP UDP 3609 +2010-12-01 23:05:01 Atlanta 10.0.0.1 53368 Texas 10.0.0.30 80 HTTP UDP 3171 +2010-12-01 23:05:02 Atlanta 10.0.0.1 51399 Texas 10.0.0.28 80 HTTP UDP 3736 +2010-12-01 23:05:03 Atlanta 10.0.0.1 50339 Texas 10.0.0.25 80 HTTP UDP 5482 +2010-12-01 23:05:04 Atlanta 10.0.0.1 64667 Texas 10.0.0.30 80 HTTP UDP 2643 +2010-12-01 23:05:05 Atlanta 10.0.0.1 51628 Texas 10.0.0.27 80 HTTP UDP 2997 +2010-12-01 23:05:06 Atlanta 10.0.0.1 52622 Texas 10.0.0.28 80 HTTP UDP 2782 +2010-12-01 23:05:07 Atlanta 10.0.0.1 55502 Texas 10.0.0.30 80 HTTP UDP 2554 +2010-12-01 23:05:08 Atlanta 10.0.0.1 57060 Texas 10.0.0.26 80 HTTP UDP 5016 +2010-12-01 23:05:09 Atlanta 10.0.0.1 50555 Texas 10.0.0.30 80 HTTP UDP 2319 +2010-12-01 23:05:10 Atlanta 10.0.0.1 59499 Texas 10.0.0.30 80 HTTP UDP 2956 +2010-12-01 23:05:11 Atlanta 10.0.0.1 56625 Texas 10.0.0.28 80 HTTP UDP 5950 +2010-12-01 23:05:12 Atlanta 10.0.0.1 56740 Texas 10.0.0.25 80 HTTP UDP 4061 +2010-12-01 23:05:13 Atlanta 10.0.0.1 58649 Texas 10.0.0.28 80 HTTP UDP 3104 +2010-12-01 23:05:14 Atlanta 10.0.0.1 62372 Texas 10.0.0.26 80 HTTP UDP 4164 +2010-12-01 23:05:15 Atlanta 10.0.0.1 53469 Texas 10.0.0.28 80 HTTP UDP 4935 +2010-12-01 23:05:16 Atlanta 10.0.0.1 64486 Texas 10.0.0.30 80 HTTP UDP 4925 +2010-12-01 23:05:17 Atlanta 10.0.0.1 52942 Texas 10.0.0.29 80 HTTP UDP 4646 +2010-12-01 23:05:18 Atlanta 10.0.0.1 63011 Texas 10.0.0.25 80 HTTP UDP 5849 +2010-12-01 23:05:19 Atlanta 10.0.0.1 61430 Texas 10.0.0.28 80 HTTP UDP 3241 +2010-12-01 23:05:20 Atlanta 10.0.0.1 59095 Texas 10.0.0.26 80 HTTP UDP 3034 +2010-12-01 23:05:21 Atlanta 10.0.0.1 62441 Texas 10.0.0.28 80 HTTP UDP 2688 +2010-12-01 23:05:22 Atlanta 10.0.0.1 51195 Texas 10.0.0.30 80 HTTP UDP 4701 +2010-12-01 23:05:23 Atlanta 10.0.0.1 57850 Texas 10.0.0.27 80 HTTP UDP 5918 +2010-12-01 23:05:24 Atlanta 10.0.0.1 50520 Texas 10.0.0.27 80 HTTP UDP 3607 +2010-12-01 23:05:25 Atlanta 10.0.0.1 59816 Texas 10.0.0.29 80 HTTP UDP 4374 +2010-12-01 23:05:26 Atlanta 10.0.0.1 58860 Texas 10.0.0.29 80 HTTP UDP 4058 +2010-12-01 23:05:27 Atlanta 10.0.0.1 55898 Texas 10.0.0.28 80 HTTP UDP 3990 +2010-12-01 23:05:28 Atlanta 10.0.0.1 51171 Texas 10.0.0.30 80 HTTP UDP 3951 +2010-12-01 23:05:29 Atlanta 10.0.0.1 61787 Texas 10.0.0.26 80 HTTP UDP 5444 +2010-12-01 23:05:30 Atlanta 10.0.0.1 61427 Texas 10.0.0.30 80 HTTP UDP 3669 +2010-12-01 23:05:31 Atlanta 10.0.0.1 65297 Texas 10.0.0.25 80 HTTP UDP 4945 +2010-12-01 23:05:32 Atlanta 10.0.0.1 61590 Texas 10.0.0.29 80 HTTP UDP 5017 +2010-12-01 23:05:33 Atlanta 10.0.0.1 55043 Texas 10.0.0.27 80 HTTP UDP 4627 +2010-12-01 23:05:34 Atlanta 10.0.0.1 53288 Texas 10.0.0.25 80 HTTP UDP 4368 +2010-12-01 23:05:35 Atlanta 10.0.0.1 64441 Texas 10.0.0.28 80 HTTP UDP 2652 +2010-12-01 23:05:36 Atlanta 10.0.0.1 62012 Texas 10.0.0.30 80 HTTP UDP 2771 +2010-12-01 23:05:37 Atlanta 10.0.0.1 60966 Texas 10.0.0.26 80 HTTP UDP 3509 +2010-12-01 23:05:38 Atlanta 10.0.0.1 52355 Texas 10.0.0.25 80 HTTP UDP 5436 +2010-12-01 23:05:39 Atlanta 10.0.0.1 65105 Texas 10.0.0.26 80 HTTP UDP 5501 +2010-12-01 23:05:40 Atlanta 10.0.0.1 65334 Texas 10.0.0.27 80 HTTP UDP 3182 +2010-12-01 23:05:41 Atlanta 10.0.0.1 56484 Texas 10.0.0.29 80 HTTP UDP 2855 +2010-12-01 23:05:42 Atlanta 10.0.0.1 65032 Texas 10.0.0.28 80 HTTP UDP 3135 +2010-12-01 23:05:43 Atlanta 10.0.0.1 65098 Texas 10.0.0.25 80 HTTP UDP 5464 +2010-12-01 23:05:44 Atlanta 10.0.0.1 55702 Texas 10.0.0.26 80 HTTP UDP 5807 +2010-12-01 23:05:45 Atlanta 10.0.0.1 52652 Texas 10.0.0.29 80 HTTP UDP 2997 +2010-12-01 23:05:46 Atlanta 10.0.0.1 60371 Texas 10.0.0.26 80 HTTP UDP 3019 +2010-12-01 23:05:47 Atlanta 10.0.0.1 51362 Texas 10.0.0.25 80 HTTP UDP 4646 +2010-12-01 23:05:48 Atlanta 10.0.0.1 61077 Texas 10.0.0.28 80 HTTP UDP 2909 +2010-12-01 23:05:49 Atlanta 10.0.0.1 53075 Texas 10.0.0.25 80 HTTP UDP 5624 +2010-12-01 23:05:50 Atlanta 10.0.0.1 50492 Texas 10.0.0.25 80 HTTP UDP 4113 +2010-12-01 23:05:51 Atlanta 10.0.0.1 49682 Texas 10.0.0.29 80 HTTP UDP 4358 +2010-12-01 23:05:52 Atlanta 10.0.0.1 54399 Texas 10.0.0.25 80 HTTP UDP 5340 +2010-12-01 23:05:53 Atlanta 10.0.0.1 59978 Texas 10.0.0.27 80 HTTP UDP 5082 +2010-12-01 23:05:54 Atlanta 10.0.0.1 54308 Texas 10.0.0.28 80 HTTP UDP 2855 +2010-12-01 23:05:55 Atlanta 10.0.0.1 65081 Texas 10.0.0.27 80 HTTP UDP 3261 +2010-12-01 23:05:56 Atlanta 10.0.0.1 54513 Texas 10.0.0.26 80 HTTP UDP 2782 +2010-12-01 23:05:57 Atlanta 10.0.0.1 58519 Texas 10.0.0.27 80 HTTP UDP 2559 +2010-12-01 23:05:58 Atlanta 10.0.0.1 63856 Texas 10.0.0.27 80 HTTP UDP 3261 +2010-12-01 23:05:59 Atlanta 10.0.0.1 52840 Texas 10.0.0.30 80 HTTP UDP 5051 +2010-12-01 23:06:00 Atlanta 10.0.0.1 59194 Texas 10.0.0.25 80 HTTP UDP 3565 +2010-12-01 23:06:01 Atlanta 10.0.0.1 53646 Texas 10.0.0.28 80 HTTP UDP 2432 +2010-12-01 23:06:02 Atlanta 10.0.0.1 60630 Texas 10.0.0.25 80 HTTP UDP 4316 +2010-12-01 23:06:03 Atlanta 10.0.0.1 61214 Texas 10.0.0.28 80 HTTP UDP 3857 +2010-12-01 23:06:04 Atlanta 10.0.0.1 54208 Texas 10.0.0.30 80 HTTP UDP 3901 +2010-12-01 23:06:05 Atlanta 10.0.0.1 52403 Texas 10.0.0.27 80 HTTP UDP 2736 +2010-12-01 23:06:06 Atlanta 10.0.0.1 59071 Texas 10.0.0.29 80 HTTP UDP 2997 +2010-12-01 23:06:07 Atlanta 10.0.0.1 63502 Texas 10.0.0.26 80 HTTP UDP 4552 +2010-12-01 23:06:08 Atlanta 10.0.0.1 63303 Texas 10.0.0.28 80 HTTP UDP 3685 +2010-12-01 23:06:09 Atlanta 10.0.0.1 57995 Texas 10.0.0.27 80 HTTP UDP 3983 +2010-12-01 23:06:10 Atlanta 10.0.0.1 65281 Texas 10.0.0.27 80 HTTP UDP 5256 +2010-12-01 23:06:11 Atlanta 10.0.0.1 50451 Texas 10.0.0.28 80 HTTP UDP 2322 +2010-12-01 23:06:12 Atlanta 10.0.0.1 58351 Texas 10.0.0.28 80 HTTP UDP 4150 +2010-12-01 23:06:13 Atlanta 10.0.0.1 50594 Texas 10.0.0.25 80 HTTP UDP 4164 +2010-12-01 23:06:14 Atlanta 10.0.0.1 56568 Texas 10.0.0.30 80 HTTP UDP 4787 +2010-12-01 23:06:15 Atlanta 10.0.0.1 60966 Texas 10.0.0.25 80 HTTP UDP 4369 +2010-12-01 23:06:16 Atlanta 10.0.0.1 50067 Texas 10.0.0.30 80 HTTP UDP 3104 +2010-12-01 23:06:17 Atlanta 10.0.0.1 50788 Texas 10.0.0.29 80 HTTP UDP 5215 +2010-12-01 23:06:18 Atlanta 10.0.0.1 56723 Texas 10.0.0.25 80 HTTP UDP 3135 +2010-12-01 23:06:19 Atlanta 10.0.0.1 62013 Texas 10.0.0.28 80 HTTP UDP 4324 +2010-12-01 23:06:20 Atlanta 10.0.0.1 65089 Texas 10.0.0.28 80 HTTP UDP 2095 +2010-12-01 23:06:21 Atlanta 10.0.0.1 50543 Texas 10.0.0.28 80 HTTP UDP 4173 +2010-12-01 23:06:22 Atlanta 10.0.0.1 65251 Texas 10.0.0.29 80 HTTP UDP 5217 +2010-12-01 23:06:23 Atlanta 10.0.0.1 63979 Texas 10.0.0.27 80 HTTP UDP 4238 +2010-12-01 23:06:24 Atlanta 10.0.0.1 53140 Texas 10.0.0.30 80 HTTP UDP 4043 +2010-12-01 23:06:25 Atlanta 10.0.0.1 58244 Texas 10.0.0.25 80 HTTP UDP 2734 +2010-12-01 23:06:26 Atlanta 10.0.0.1 56245 Texas 10.0.0.26 80 HTTP UDP 5340 +2010-12-01 23:06:27 Atlanta 10.0.0.1 64975 Texas 10.0.0.28 80 HTTP UDP 2003 +2010-12-01 23:06:28 Atlanta 10.0.0.1 57433 Texas 10.0.0.27 80 HTTP UDP 5935 +2010-12-01 23:06:29 Atlanta 10.0.0.1 51247 Texas 10.0.0.26 80 HTTP UDP 2875 +2010-12-01 23:06:30 Atlanta 10.0.0.1 51927 Texas 10.0.0.29 80 HTTP UDP 3499 +2010-12-01 23:06:31 Atlanta 10.0.0.1 51807 Texas 10.0.0.28 80 HTTP UDP 3054 +2010-12-01 23:06:32 Atlanta 10.0.0.1 56132 Texas 10.0.0.27 80 HTTP UDP 3171 +2010-12-01 23:06:33 Atlanta 10.0.0.1 57036 Texas 10.0.0.30 80 HTTP UDP 4735 +2010-12-01 23:06:34 Atlanta 10.0.0.1 57947 Texas 10.0.0.30 80 HTTP UDP 4515 +2010-12-01 23:06:35 Atlanta 10.0.0.1 54090 Texas 10.0.0.29 80 HTTP UDP 3174 +2010-12-01 23:06:36 Atlanta 10.0.0.1 55831 Texas 10.0.0.25 80 HTTP UDP 5346 +2010-12-01 23:06:37 Atlanta 10.0.0.1 49465 Texas 10.0.0.28 80 HTTP UDP 2022 +2010-12-01 23:06:38 Atlanta 10.0.0.1 54465 Texas 10.0.0.30 80 HTTP UDP 3267 +2010-12-01 23:06:39 Atlanta 10.0.0.1 55638 Texas 10.0.0.26 80 HTTP UDP 4864 +2010-12-01 23:06:40 Atlanta 10.0.0.1 52185 Texas 10.0.0.26 80 HTTP UDP 2836 +2010-12-01 23:06:41 Atlanta 10.0.0.1 63786 Texas 10.0.0.29 80 HTTP UDP 5480 +2010-12-01 23:06:42 Atlanta 10.0.0.1 65375 Texas 10.0.0.26 80 HTTP UDP 4395 +2010-12-01 23:06:43 Atlanta 10.0.0.1 49685 Texas 10.0.0.29 80 HTTP UDP 2603 +2010-12-01 23:06:44 Atlanta 10.0.0.1 51996 Texas 10.0.0.29 80 HTTP UDP 5422 +2010-12-01 23:06:45 Atlanta 10.0.0.1 54709 Texas 10.0.0.30 80 HTTP UDP 4711 +2010-12-01 23:06:46 Atlanta 10.0.0.1 61588 Texas 10.0.0.29 80 HTTP UDP 4427 +2010-12-01 23:06:47 Atlanta 10.0.0.1 52799 Texas 10.0.0.29 80 HTTP UDP 3568 +2010-12-01 23:06:48 Atlanta 10.0.0.1 65057 Texas 10.0.0.25 80 HTTP UDP 2235 +2010-12-01 23:06:49 Atlanta 10.0.0.1 49521 Texas 10.0.0.26 80 HTTP UDP 2014 +2010-12-01 23:06:50 Atlanta 10.0.0.1 60208 Texas 10.0.0.29 80 HTTP UDP 2273 +2010-12-01 23:06:51 Atlanta 10.0.0.1 57949 Texas 10.0.0.25 80 HTTP UDP 2038 +2010-12-01 23:06:52 Atlanta 10.0.0.1 64227 Texas 10.0.0.28 80 HTTP UDP 3987 +2010-12-01 23:06:53 Atlanta 10.0.0.1 56201 Texas 10.0.0.29 80 HTTP UDP 2554 +2010-12-01 23:06:54 Atlanta 10.0.0.1 63823 Texas 10.0.0.30 80 HTTP UDP 3657 +2010-12-01 23:06:55 Atlanta 10.0.0.1 58321 Texas 10.0.0.25 80 HTTP UDP 3708 +2010-12-01 23:06:56 Atlanta 10.0.0.1 63907 Texas 10.0.0.28 80 HTTP UDP 3761 +2010-12-01 23:06:57 Atlanta 10.0.0.1 57580 Texas 10.0.0.25 80 HTTP UDP 2602 +2010-12-01 23:06:58 Atlanta 10.0.0.1 62022 Texas 10.0.0.28 80 HTTP UDP 2361 +2010-12-01 23:06:59 Atlanta 10.0.0.1 63712 Texas 10.0.0.28 80 HTTP UDP 5020 +2010-12-01 23:07:00 Atlanta 10.0.0.1 61638 Texas 10.0.0.26 80 HTTP UDP 3729 +2010-12-01 23:07:01 Atlanta 10.0.0.1 49413 Texas 10.0.0.26 80 HTTP UDP 3063 +2010-12-01 23:07:02 Atlanta 10.0.0.1 59663 Texas 10.0.0.25 80 HTTP UDP 5684 +2010-12-01 23:07:03 Atlanta 10.0.0.1 60844 Texas 10.0.0.28 80 HTTP UDP 5015 +2010-12-01 23:07:04 Atlanta 10.0.0.1 59145 Texas 10.0.0.28 80 HTTP UDP 3671 +2010-12-01 23:07:05 Atlanta 10.0.0.1 59484 Texas 10.0.0.25 80 HTTP UDP 2906 +2010-12-01 23:07:06 Atlanta 10.0.0.1 54032 Texas 10.0.0.28 80 HTTP UDP 3581 +2010-12-01 23:07:07 Atlanta 10.0.0.1 58024 Texas 10.0.0.28 80 HTTP UDP 3568 +2010-12-01 23:07:08 Atlanta 10.0.0.1 55356 Texas 10.0.0.26 80 HTTP UDP 4903 +2010-12-01 23:07:09 Atlanta 10.0.0.1 54808 Texas 10.0.0.30 80 HTTP UDP 2757 +2010-12-01 23:07:10 Atlanta 10.0.0.1 49730 Texas 10.0.0.28 80 HTTP UDP 4735 +2010-12-01 23:07:11 Atlanta 10.0.0.1 51309 Texas 10.0.0.27 80 HTTP UDP 2997 +2010-12-01 23:07:12 Atlanta 10.0.0.1 64937 Texas 10.0.0.25 80 HTTP UDP 3135 +2010-12-01 23:07:13 Atlanta 10.0.0.1 54215 Texas 10.0.0.26 80 HTTP UDP 5348 +2010-12-01 23:07:14 Atlanta 10.0.0.1 58009 Texas 10.0.0.28 80 HTTP UDP 2221 +2010-12-01 23:07:15 Atlanta 10.0.0.1 59377 Texas 10.0.0.27 80 HTTP UDP 5313 +2010-12-01 23:07:16 Atlanta 10.0.0.1 50098 Texas 10.0.0.29 80 HTTP UDP 3860 +2010-12-01 23:07:17 Atlanta 10.0.0.1 57778 Texas 10.0.0.25 80 HTTP UDP 5092 +2010-12-01 23:07:18 Atlanta 10.0.0.1 59966 Texas 10.0.0.25 80 HTTP UDP 5700 +2010-12-01 23:07:19 Atlanta 10.0.0.1 60024 Texas 10.0.0.25 80 HTTP UDP 3864 +2010-12-01 23:07:20 Atlanta 10.0.0.1 62249 Texas 10.0.0.29 80 HTTP UDP 5884 +2010-12-01 23:07:21 Atlanta 10.0.0.1 62345 Texas 10.0.0.30 80 HTTP UDP 2965 +2010-12-01 23:07:22 Atlanta 10.0.0.1 56257 Texas 10.0.0.25 80 HTTP UDP 2947 +2010-12-01 23:07:23 Atlanta 10.0.0.1 52306 Texas 10.0.0.28 80 HTTP UDP 4097 +2010-12-01 23:07:24 Atlanta 10.0.0.1 62922 Texas 10.0.0.26 80 HTTP UDP 4324 +2010-12-01 23:07:25 Atlanta 10.0.0.1 58820 Texas 10.0.0.26 80 HTTP UDP 4061 +2010-12-01 23:07:26 Atlanta 10.0.0.1 53277 Texas 10.0.0.28 80 HTTP UDP 2432 +2010-12-01 23:07:27 Atlanta 10.0.0.1 65349 Texas 10.0.0.25 80 HTTP UDP 2954 +2010-12-01 23:07:28 Atlanta 10.0.0.1 61793 Texas 10.0.0.25 80 HTTP UDP 2924 +2010-12-01 23:07:29 Atlanta 10.0.0.1 49333 Texas 10.0.0.27 80 HTTP UDP 5919 +2010-12-01 23:07:30 Atlanta 10.0.0.1 59506 Texas 10.0.0.29 80 HTTP UDP 4757 +2010-12-01 23:07:31 Atlanta 10.0.0.1 62526 Texas 10.0.0.28 80 HTTP UDP 5867 +2010-12-01 23:07:32 Atlanta 10.0.0.1 54990 Texas 10.0.0.28 80 HTTP UDP 5963 +2010-12-01 23:07:33 Atlanta 10.0.0.1 50009 Texas 10.0.0.26 80 HTTP UDP 4744 +2010-12-01 23:07:34 Atlanta 10.0.0.1 65425 Texas 10.0.0.26 80 HTTP UDP 5696 +2010-12-01 23:07:35 Atlanta 10.0.0.1 58700 Texas 10.0.0.28 80 HTTP UDP 5556 +2010-12-01 23:07:36 Atlanta 10.0.0.1 52909 Texas 10.0.0.27 80 HTTP UDP 5853 +2010-12-01 23:07:37 Atlanta 10.0.0.1 64898 Texas 10.0.0.26 80 HTTP UDP 2906 +2010-12-01 23:07:38 Atlanta 10.0.0.1 58242 Texas 10.0.0.29 80 HTTP UDP 3146 +2010-12-01 23:07:39 Atlanta 10.0.0.1 64863 Texas 10.0.0.27 80 HTTP UDP 3107 +2010-12-01 23:07:40 Atlanta 10.0.0.1 54369 Texas 10.0.0.26 80 HTTP UDP 4506 +2010-12-01 23:07:41 Atlanta 10.0.0.1 60430 Texas 10.0.0.26 80 HTTP UDP 5252 +2010-12-01 23:07:42 Atlanta 10.0.0.1 52486 Texas 10.0.0.25 80 HTTP UDP 4787 +2010-12-01 23:07:43 Atlanta 10.0.0.1 55942 Texas 10.0.0.25 80 HTTP UDP 2860 +2010-12-01 23:07:44 Atlanta 10.0.0.1 63485 Texas 10.0.0.30 80 HTTP UDP 5696 +2010-12-01 23:07:45 Atlanta 10.0.0.1 64595 Texas 10.0.0.26 80 HTTP UDP 2172 +2010-12-01 23:07:46 Atlanta 10.0.0.1 52467 Texas 10.0.0.28 80 HTTP UDP 4589 +2010-12-01 23:07:47 Atlanta 10.0.0.1 64576 Texas 10.0.0.30 80 HTTP UDP 5989 +2010-12-01 23:07:48 Atlanta 10.0.0.1 62622 Texas 10.0.0.27 80 HTTP UDP 5605 +2010-12-01 23:07:49 Atlanta 10.0.0.1 50824 Texas 10.0.0.30 80 HTTP UDP 2114 +2010-12-01 23:07:50 Atlanta 10.0.0.1 58630 Texas 10.0.0.29 80 HTTP UDP 4273 +2010-12-01 23:07:51 Atlanta 10.0.0.1 62591 Texas 10.0.0.30 80 HTTP UDP 5694 +2010-12-01 23:07:52 Atlanta 10.0.0.1 52749 Texas 10.0.0.26 80 HTTP UDP 3918 +2010-12-01 23:07:53 Atlanta 10.0.0.1 64616 Texas 10.0.0.30 80 HTTP UDP 5273 +2010-12-01 23:07:54 Atlanta 10.0.0.1 55003 Texas 10.0.0.25 80 HTTP UDP 4871 +2010-12-01 23:07:55 Atlanta 10.0.0.1 54766 Texas 10.0.0.27 80 HTTP UDP 5135 +2010-12-01 23:07:56 Atlanta 10.0.0.1 52362 Texas 10.0.0.30 80 HTTP UDP 2432 +2010-12-01 23:07:57 Atlanta 10.0.0.1 61953 Texas 10.0.0.27 80 HTTP UDP 3806 +2010-12-01 23:07:58 Atlanta 10.0.0.1 62375 Texas 10.0.0.29 80 HTTP UDP 4724 +2010-12-01 23:07:59 Atlanta 10.0.0.1 55055 Texas 10.0.0.27 80 HTTP UDP 3174 +2010-12-01 23:08:00 Atlanta 10.0.0.1 58905 Texas 10.0.0.28 80 HTTP UDP 3156 +2010-12-01 23:08:01 Atlanta 10.0.0.1 49977 Texas 10.0.0.27 80 HTTP UDP 2545 +2010-12-01 23:08:02 Atlanta 10.0.0.1 58690 Texas 10.0.0.26 80 HTTP UDP 2561 +2010-12-01 23:08:03 Atlanta 10.0.0.1 50334 Texas 10.0.0.25 80 HTTP UDP 3104 +2010-12-01 23:08:04 Atlanta 10.0.0.1 62902 Texas 10.0.0.26 80 HTTP UDP 4121 +2010-12-01 23:08:05 Atlanta 10.0.0.1 52397 Texas 10.0.0.26 80 HTTP UDP 3009 +2010-12-01 23:08:06 Atlanta 10.0.0.1 54771 Texas 10.0.0.27 80 HTTP UDP 4692 +2010-12-01 23:08:07 Atlanta 10.0.0.1 56068 Texas 10.0.0.25 80 HTTP UDP 3512 +2010-12-01 23:08:08 Atlanta 10.0.0.1 65110 Texas 10.0.0.29 80 HTTP UDP 4934 +2010-12-01 23:08:09 Atlanta 10.0.0.1 63179 Texas 10.0.0.27 80 HTTP UDP 2519 +2010-12-01 23:08:10 Atlanta 10.0.0.1 57495 Texas 10.0.0.29 80 HTTP UDP 4751 +2010-12-01 23:08:11 Atlanta 10.0.0.1 51044 Texas 10.0.0.26 80 HTTP UDP 2701 +2010-12-01 23:08:12 Atlanta 10.0.0.1 65271 Texas 10.0.0.25 80 HTTP UDP 5366 +2010-12-01 23:08:13 Atlanta 10.0.0.1 57385 Texas 10.0.0.30 80 HTTP UDP 2102 +2010-12-01 23:08:14 Atlanta 10.0.0.1 56505 Texas 10.0.0.25 80 HTTP UDP 2432 +2010-12-01 23:08:15 Atlanta 10.0.0.1 59050 Texas 10.0.0.29 80 HTTP UDP 2380 +2010-12-01 23:08:16 Atlanta 10.0.0.1 63903 Texas 10.0.0.29 80 HTTP UDP 3499 +2010-12-01 23:08:17 Atlanta 10.0.0.1 60302 Texas 10.0.0.29 80 HTTP UDP 2240 +2010-12-01 23:08:18 Atlanta 10.0.0.1 55075 Texas 10.0.0.30 80 HTTP UDP 5132 +2010-12-01 23:08:19 Atlanta 10.0.0.1 49220 Texas 10.0.0.29 80 HTTP UDP 5562 +2010-12-01 23:08:20 Atlanta 10.0.0.1 64054 Texas 10.0.0.30 80 HTTP UDP 3540 +2010-12-01 23:08:21 Atlanta 10.0.0.1 49688 Texas 10.0.0.26 80 HTTP UDP 5039 +2010-12-01 23:08:22 Atlanta 10.0.0.1 57569 Texas 10.0.0.29 80 HTTP UDP 5935 +2010-12-01 23:08:23 Atlanta 10.0.0.1 63589 Texas 10.0.0.28 80 HTTP UDP 4757 +2010-12-01 23:08:24 Atlanta 10.0.0.1 60679 Texas 10.0.0.26 80 HTTP UDP 2242 +2010-12-01 23:08:25 Atlanta 10.0.0.1 56497 Texas 10.0.0.28 80 HTTP UDP 2836 +2010-12-01 23:08:26 Atlanta 10.0.0.1 55939 Texas 10.0.0.28 80 HTTP UDP 2168 +2010-12-01 23:08:27 Atlanta 10.0.0.1 50794 Texas 10.0.0.28 80 HTTP UDP 4195 +2010-12-01 23:08:28 Atlanta 10.0.0.1 59089 Texas 10.0.0.26 80 HTTP UDP 4006 +2010-12-01 23:08:29 Atlanta 10.0.0.1 59969 Texas 10.0.0.25 80 HTTP UDP 4369 +2010-12-01 23:08:30 Atlanta 10.0.0.1 60562 Texas 10.0.0.29 80 HTTP UDP 5527 +2010-12-01 23:08:31 Atlanta 10.0.0.1 58805 Texas 10.0.0.25 80 HTTP UDP 2098 +2010-12-01 23:08:32 Atlanta 10.0.0.1 59990 Texas 10.0.0.29 80 HTTP UDP 2690 +2010-12-01 23:08:33 Atlanta 10.0.0.1 55244 Texas 10.0.0.25 80 HTTP UDP 5696 +2010-12-01 23:08:34 Atlanta 10.0.0.1 61810 Texas 10.0.0.30 80 HTTP UDP 4013 +2010-12-01 23:08:35 Atlanta 10.0.0.1 55868 Texas 10.0.0.28 80 HTTP UDP 2602 +2010-12-01 23:08:36 Atlanta 10.0.0.1 61674 Texas 10.0.0.29 80 HTTP UDP 3806 +2010-12-01 23:08:37 Atlanta 10.0.0.1 56388 Texas 10.0.0.26 80 HTTP UDP 3905 +2010-12-01 23:08:38 Atlanta 10.0.0.1 62472 Texas 10.0.0.27 80 HTTP UDP 4868 +2010-12-01 23:08:39 Atlanta 10.0.0.1 63304 Texas 10.0.0.26 80 HTTP UDP 3934 +2010-12-01 23:08:40 Atlanta 10.0.0.1 55791 Texas 10.0.0.28 80 HTTP UDP 3793 +2010-12-01 23:08:41 Atlanta 10.0.0.1 60203 Texas 10.0.0.30 80 HTTP UDP 4711 +2010-12-01 23:08:42 Atlanta 10.0.0.1 62761 Texas 10.0.0.28 80 HTTP UDP 4189 +2010-12-01 23:08:43 Atlanta 10.0.0.1 52394 Texas 10.0.0.27 80 HTTP UDP 3431 +2010-12-01 23:08:44 Atlanta 10.0.0.1 58409 Texas 10.0.0.26 80 HTTP UDP 4948 +2010-12-01 23:08:45 Atlanta 10.0.0.1 56644 Texas 10.0.0.30 80 HTTP UDP 5542 +2010-12-01 23:08:46 Atlanta 10.0.0.1 55337 Texas 10.0.0.25 80 HTTP UDP 5072 +2010-12-01 23:08:47 Atlanta 10.0.0.1 64523 Texas 10.0.0.30 80 HTTP UDP 3742 +2010-12-01 23:08:48 Atlanta 10.0.0.1 63894 Texas 10.0.0.26 80 HTTP UDP 2085 +2010-12-01 23:08:49 Atlanta 10.0.0.1 51683 Texas 10.0.0.28 80 HTTP UDP 3996 +2010-12-01 23:08:50 Atlanta 10.0.0.1 52498 Texas 10.0.0.28 80 HTTP UDP 4587 +2010-12-01 23:08:51 Atlanta 10.0.0.1 51108 Texas 10.0.0.25 80 HTTP UDP 4058 +2010-12-01 23:08:52 Atlanta 10.0.0.1 49210 Texas 10.0.0.29 80 HTTP UDP 4774 +2010-12-01 23:08:53 Atlanta 10.0.0.1 58835 Texas 10.0.0.30 80 HTTP UDP 4281 +2010-12-01 23:08:54 Atlanta 10.0.0.1 49836 Texas 10.0.0.26 80 HTTP UDP 3138 +2010-12-01 23:08:55 Atlanta 10.0.0.1 51059 Texas 10.0.0.25 80 HTTP UDP 3628 +2010-12-01 23:08:56 Atlanta 10.0.0.1 54787 Texas 10.0.0.30 80 HTTP UDP 4399 +2010-12-01 23:08:57 Atlanta 10.0.0.1 51426 Texas 10.0.0.26 80 HTTP UDP 3079 +2010-12-01 23:08:58 Atlanta 10.0.0.1 54424 Texas 10.0.0.27 80 HTTP UDP 2107 +2010-12-01 23:08:59 Atlanta 10.0.0.1 60877 Texas 10.0.0.30 80 HTTP UDP 4565 +2010-12-01 23:09:00 Atlanta 10.0.0.1 60126 Texas 10.0.0.27 80 HTTP UDP 3733 +2010-12-01 23:09:01 Atlanta 10.0.0.1 63501 Texas 10.0.0.28 80 HTTP UDP 5483 +2010-12-01 23:09:02 Atlanta 10.0.0.1 62241 Texas 10.0.0.26 80 HTTP UDP 5397 +2010-12-01 23:09:03 Atlanta 10.0.0.1 59141 Texas 10.0.0.26 80 HTTP UDP 4669 +2010-12-01 23:09:04 Atlanta 10.0.0.1 63892 Texas 10.0.0.30 80 HTTP UDP 2523 +2010-12-01 23:09:05 Atlanta 10.0.0.1 49718 Texas 10.0.0.28 80 HTTP UDP 5004 +2010-12-01 23:09:06 Atlanta 10.0.0.1 49818 Texas 10.0.0.30 80 HTTP UDP 5057 +2010-12-01 23:09:07 Atlanta 10.0.0.1 62760 Texas 10.0.0.28 80 HTTP UDP 3450 +2010-12-01 23:09:08 Atlanta 10.0.0.1 55648 Texas 10.0.0.29 80 HTTP UDP 2642 +2010-12-01 23:09:09 Atlanta 10.0.0.1 62233 Texas 10.0.0.27 80 HTTP UDP 4270 +2010-12-01 23:09:10 Atlanta 10.0.0.1 54071 Texas 10.0.0.27 80 HTTP UDP 5989 +2010-12-01 23:09:11 Atlanta 10.0.0.1 65139 Texas 10.0.0.29 80 HTTP UDP 3292 +2010-12-01 23:09:12 Atlanta 10.0.0.1 49501 Texas 10.0.0.26 80 HTTP UDP 2689 +2010-12-01 23:09:13 Atlanta 10.0.0.1 62568 Texas 10.0.0.27 80 HTTP UDP 5558 +2010-12-01 23:09:14 Atlanta 10.0.0.1 57108 Texas 10.0.0.25 80 HTTP UDP 4164 +2010-12-01 23:09:15 Atlanta 10.0.0.1 52688 Texas 10.0.0.28 80 HTTP UDP 2910 +2010-12-01 23:09:16 Atlanta 10.0.0.1 56415 Texas 10.0.0.30 80 HTTP UDP 4319 +2010-12-01 23:09:17 Atlanta 10.0.0.1 65132 Texas 10.0.0.29 80 HTTP UDP 3067 +2010-12-01 23:09:18 Atlanta 10.0.0.1 59956 Texas 10.0.0.27 80 HTTP UDP 3285 +2010-12-01 23:09:19 Atlanta 10.0.0.1 61977 Texas 10.0.0.30 80 HTTP UDP 2784 +2010-12-01 23:09:20 Atlanta 10.0.0.1 55033 Texas 10.0.0.26 80 HTTP UDP 4113 +2010-12-01 23:09:21 Atlanta 10.0.0.1 62006 Texas 10.0.0.25 80 HTTP UDP 5570 +2010-12-01 23:09:22 Atlanta 10.0.0.1 54450 Texas 10.0.0.26 80 HTTP UDP 4918 +2010-12-01 23:09:23 Atlanta 10.0.0.1 57788 Texas 10.0.0.28 80 HTTP UDP 2308 +2010-12-01 23:09:24 Atlanta 10.0.0.1 62675 Texas 10.0.0.28 80 HTTP UDP 3732 +2010-12-01 23:09:25 Atlanta 10.0.0.1 54725 Texas 10.0.0.26 80 HTTP UDP 3318 +2010-12-01 23:09:26 Atlanta 10.0.0.1 63371 Texas 10.0.0.25 80 HTTP UDP 3758 +2010-12-01 23:09:27 Atlanta 10.0.0.1 53456 Texas 10.0.0.28 80 HTTP UDP 4903 +2010-12-01 23:09:28 Atlanta 10.0.0.1 54736 Texas 10.0.0.26 80 HTTP UDP 2249 +2010-12-01 23:09:29 Atlanta 10.0.0.1 60668 Texas 10.0.0.28 80 HTTP UDP 4327 +2010-12-01 23:09:30 Atlanta 10.0.0.1 56542 Texas 10.0.0.25 80 HTTP UDP 5412 +2010-12-01 23:09:31 Atlanta 10.0.0.1 54492 Texas 10.0.0.26 80 HTTP UDP 3107 +2010-12-01 23:09:32 Atlanta 10.0.0.1 61234 Texas 10.0.0.26 80 HTTP UDP 2144 +2010-12-01 23:09:33 Atlanta 10.0.0.1 56671 Texas 10.0.0.25 80 HTTP UDP 2732 +2010-12-01 23:09:34 Atlanta 10.0.0.1 63832 Texas 10.0.0.26 80 HTTP UDP 5422 +2010-12-01 23:09:35 Atlanta 10.0.0.1 53840 Texas 10.0.0.26 80 HTTP UDP 3019 +2010-12-01 23:09:36 Atlanta 10.0.0.1 59724 Texas 10.0.0.29 80 HTTP UDP 5110 +2010-12-01 23:09:37 Atlanta 10.0.0.1 60550 Texas 10.0.0.30 80 HTTP UDP 4088 +2010-12-01 23:09:38 Atlanta 10.0.0.1 64054 Texas 10.0.0.30 80 HTTP UDP 5347 +2010-12-01 23:09:39 Atlanta 10.0.0.1 58993 Texas 10.0.0.26 80 HTTP UDP 5624 +2010-12-01 23:09:40 Atlanta 10.0.0.1 55392 Texas 10.0.0.25 80 HTTP UDP 5997 +2010-12-01 23:09:41 Atlanta 10.0.0.1 64465 Texas 10.0.0.28 80 HTTP UDP 3125 +2010-12-01 23:09:42 Atlanta 10.0.0.1 56949 Texas 10.0.0.27 80 HTTP UDP 3501 +2010-12-01 23:09:43 Atlanta 10.0.0.1 50030 Texas 10.0.0.26 80 HTTP UDP 3224 +2010-12-01 23:09:44 Atlanta 10.0.0.1 52241 Texas 10.0.0.30 80 HTTP UDP 5460 +2010-12-01 23:09:45 Atlanta 10.0.0.1 57322 Texas 10.0.0.26 80 HTTP UDP 4724 +2010-12-01 23:09:46 Atlanta 10.0.0.1 55554 Texas 10.0.0.27 80 HTTP UDP 4316 +2010-12-01 23:09:47 Atlanta 10.0.0.1 57725 Texas 10.0.0.29 80 HTTP UDP 5336 +2010-12-01 23:09:48 Atlanta 10.0.0.1 51282 Texas 10.0.0.28 80 HTTP UDP 2523 +2010-12-01 23:09:49 Atlanta 10.0.0.1 52509 Texas 10.0.0.25 80 HTTP UDP 4918 +2010-12-01 23:09:50 Atlanta 10.0.0.1 62297 Texas 10.0.0.26 80 HTTP UDP 2240 +2010-12-01 23:09:51 Atlanta 10.0.0.1 61049 Texas 10.0.0.25 80 HTTP UDP 4040 +2010-12-01 23:09:52 Atlanta 10.0.0.1 55462 Texas 10.0.0.28 80 HTTP UDP 4750 +2010-12-01 23:09:53 Atlanta 10.0.0.1 53980 Texas 10.0.0.27 80 HTTP UDP 5225 +2010-12-01 23:09:54 Atlanta 10.0.0.1 50045 Texas 10.0.0.29 80 HTTP UDP 5160 +2010-12-01 23:09:55 Atlanta 10.0.0.1 64428 Texas 10.0.0.30 80 HTTP UDP 2189 +2010-12-01 23:09:56 Atlanta 10.0.0.1 52009 Texas 10.0.0.27 80 HTTP UDP 2664 +2010-12-01 23:09:57 Atlanta 10.0.0.1 58737 Texas 10.0.0.25 80 HTTP UDP 4399 +2010-12-01 23:09:58 Atlanta 10.0.0.1 59687 Texas 10.0.0.25 80 HTTP UDP 3096 +2010-12-01 23:09:59 Atlanta 10.0.0.1 60918 Texas 10.0.0.28 80 HTTP UDP 4521 +2010-12-01 23:10:00 Atlanta 10.0.0.1 53060 Texas 10.0.0.26 80 HTTP UDP 3318 +2010-12-01 23:10:01 Atlanta 10.0.0.1 59730 Texas 10.0.0.28 80 HTTP UDP 3905 +2010-12-01 23:10:02 Atlanta 10.0.0.1 56382 Texas 10.0.0.25 80 HTTP UDP 3628 +2010-12-01 23:10:03 Atlanta 10.0.0.1 51328 Texas 10.0.0.29 80 HTTP UDP 2684 +2010-12-01 23:10:04 Atlanta 10.0.0.1 52843 Texas 10.0.0.25 80 HTTP UDP 5340 +2010-12-01 23:10:05 Atlanta 10.0.0.1 49194 Texas 10.0.0.30 80 HTTP UDP 5673 +2010-12-01 23:10:06 Atlanta 10.0.0.1 60907 Texas 10.0.0.30 80 HTTP UDP 4328 +2010-12-01 23:10:07 Atlanta 10.0.0.1 65274 Texas 10.0.0.26 80 HTTP UDP 5422 +2010-12-01 23:10:08 Atlanta 10.0.0.1 57918 Texas 10.0.0.29 80 HTTP UDP 5092 +2010-12-01 23:10:09 Atlanta 10.0.0.1 54541 Texas 10.0.0.27 80 HTTP UDP 3136 +2010-12-01 23:10:10 Atlanta 10.0.0.1 64958 Texas 10.0.0.29 80 HTTP UDP 5804 +2010-12-01 23:10:11 Atlanta 10.0.0.1 65368 Texas 10.0.0.27 80 HTTP UDP 3476 +2010-12-01 23:10:12 Atlanta 10.0.0.1 64759 Texas 10.0.0.27 80 HTTP UDP 4420 +2010-12-01 23:10:13 Atlanta 10.0.0.1 59047 Texas 10.0.0.26 80 HTTP UDP 3736 +2010-12-01 23:10:14 Atlanta 10.0.0.1 52301 Texas 10.0.0.29 80 HTTP UDP 5527 +2010-12-01 23:10:15 Atlanta 10.0.0.1 63895 Texas 10.0.0.30 80 HTTP UDP 2498 +2010-12-01 23:10:16 Atlanta 10.0.0.1 56224 Texas 10.0.0.28 80 HTTP UDP 3535 +2010-12-01 23:10:17 Atlanta 10.0.0.1 63772 Texas 10.0.0.28 80 HTTP UDP 5548 +2010-12-01 23:10:18 Atlanta 10.0.0.1 53237 Texas 10.0.0.30 80 HTTP UDP 2579 +2010-12-01 23:10:19 Atlanta 10.0.0.1 59504 Texas 10.0.0.28 80 HTTP UDP 2651 +2010-12-01 23:10:20 Atlanta 10.0.0.1 60292 Texas 10.0.0.29 80 HTTP UDP 3708 +2010-12-01 23:10:21 Atlanta 10.0.0.1 55285 Texas 10.0.0.28 80 HTTP UDP 4478 +2010-12-01 23:10:22 Atlanta 10.0.0.1 65440 Texas 10.0.0.27 80 HTTP UDP 3074 +2010-12-01 23:10:23 Atlanta 10.0.0.1 61308 Texas 10.0.0.30 80 HTTP UDP 3136 +2010-12-01 23:10:24 Atlanta 10.0.0.1 53288 Texas 10.0.0.30 80 HTTP UDP 5634 +2010-12-01 23:10:25 Atlanta 10.0.0.1 55834 Texas 10.0.0.29 80 HTTP UDP 4750 +2010-12-01 23:10:26 Atlanta 10.0.0.1 55597 Texas 10.0.0.27 80 HTTP UDP 2782 +2010-12-01 23:10:27 Atlanta 10.0.0.1 55680 Texas 10.0.0.29 80 HTTP UDP 5917 +2010-12-01 23:10:28 Atlanta 10.0.0.1 57886 Texas 10.0.0.30 80 HTTP UDP 4449 +2010-12-01 23:10:29 Atlanta 10.0.0.1 64639 Texas 10.0.0.30 80 HTTP UDP 5029 +2010-12-01 23:10:30 Atlanta 10.0.0.1 61930 Texas 10.0.0.30 80 HTTP UDP 2380 +2010-12-01 23:10:31 Atlanta 10.0.0.1 58931 Texas 10.0.0.25 80 HTTP UDP 5343 +2010-12-01 23:10:32 Atlanta 10.0.0.1 61856 Texas 10.0.0.28 80 HTTP UDP 3063 +2010-12-01 23:10:33 Atlanta 10.0.0.1 51465 Texas 10.0.0.25 80 HTTP UDP 4552 +2010-12-01 23:10:34 Atlanta 10.0.0.1 58677 Texas 10.0.0.26 80 HTTP UDP 3673 +2010-12-01 23:10:35 Atlanta 10.0.0.1 55260 Texas 10.0.0.28 80 HTTP UDP 5162 +2010-12-01 23:10:36 Atlanta 10.0.0.1 60747 Texas 10.0.0.25 80 HTTP UDP 2046 +2010-12-01 23:10:37 Atlanta 10.0.0.1 60166 Texas 10.0.0.27 80 HTTP UDP 5584 +2010-12-01 23:10:38 Atlanta 10.0.0.1 58994 Texas 10.0.0.28 80 HTTP UDP 5749 +2010-12-01 23:10:39 Atlanta 10.0.0.1 50890 Texas 10.0.0.30 80 HTTP UDP 3549 +2010-12-01 23:10:40 Atlanta 10.0.0.1 61046 Texas 10.0.0.26 80 HTTP UDP 3935 +2010-12-01 23:10:41 Atlanta 10.0.0.1 59303 Texas 10.0.0.29 80 HTTP UDP 5015 +2010-12-01 23:10:42 Atlanta 10.0.0.1 53995 Texas 10.0.0.28 80 HTTP UDP 2273 +2010-12-01 23:10:43 Atlanta 10.0.0.1 61000 Texas 10.0.0.26 80 HTTP UDP 5907 +2010-12-01 23:10:44 Atlanta 10.0.0.1 52439 Texas 10.0.0.29 80 HTTP UDP 5211 +2010-12-01 23:10:45 Atlanta 10.0.0.1 55079 Texas 10.0.0.28 80 HTTP UDP 5559 +2010-12-01 23:10:46 Atlanta 10.0.0.1 59002 Texas 10.0.0.26 80 HTTP UDP 4273 +2010-12-01 23:10:47 Atlanta 10.0.0.1 56915 Texas 10.0.0.30 80 HTTP UDP 2178 +2010-12-01 23:10:48 Atlanta 10.0.0.1 61363 Texas 10.0.0.29 80 HTTP UDP 3628 +2010-12-01 23:10:49 Atlanta 10.0.0.1 58951 Texas 10.0.0.29 80 HTTP UDP 2664 +2010-12-01 23:10:50 Atlanta 10.0.0.1 64052 Texas 10.0.0.30 80 HTTP UDP 3729 +2010-12-01 23:10:51 Atlanta 10.0.0.1 55970 Texas 10.0.0.27 80 HTTP UDP 4399 +2010-12-01 23:10:52 Atlanta 10.0.0.1 57146 Texas 10.0.0.26 80 HTTP UDP 2822 +2010-12-01 23:10:53 Atlanta 10.0.0.1 61422 Texas 10.0.0.28 80 HTTP UDP 2732 +2010-12-01 23:10:54 Atlanta 10.0.0.1 53531 Texas 10.0.0.30 80 HTTP UDP 5047 +2010-12-01 23:10:55 Atlanta 10.0.0.1 64997 Texas 10.0.0.26 80 HTTP UDP 4013 +2010-12-01 23:10:56 Atlanta 10.0.0.1 49792 Texas 10.0.0.26 80 HTTP UDP 4647 +2010-12-01 23:10:57 Atlanta 10.0.0.1 55789 Texas 10.0.0.25 80 HTTP UDP 2602 +2010-12-01 23:10:58 Atlanta 10.0.0.1 60654 Texas 10.0.0.27 80 HTTP UDP 4414 +2010-12-01 23:10:59 Atlanta 10.0.0.1 51196 Texas 10.0.0.28 80 HTTP UDP 5412 +2010-12-01 23:11:00 Atlanta 10.0.0.1 62816 Texas 10.0.0.26 80 HTTP UDP 5850 +2010-12-01 23:11:01 Atlanta 10.0.0.1 59589 Texas 10.0.0.29 80 HTTP UDP 4587 +2010-12-01 23:11:02 Atlanta 10.0.0.1 63564 Texas 10.0.0.28 80 HTTP UDP 5004 +2010-12-01 23:11:03 Atlanta 10.0.0.1 64329 Texas 10.0.0.28 80 HTTP UDP 2818 +2010-12-01 23:11:04 Atlanta 10.0.0.1 55433 Texas 10.0.0.26 80 HTTP UDP 3905 +2010-12-01 23:11:05 Atlanta 10.0.0.1 55951 Texas 10.0.0.30 80 HTTP UDP 5191 +2010-12-01 23:11:06 Atlanta 10.0.0.1 56213 Texas 10.0.0.25 80 HTTP UDP 5484 +2010-12-01 23:11:07 Atlanta 10.0.0.1 65282 Texas 10.0.0.29 80 HTTP UDP 3138 +2010-12-01 23:11:08 Atlanta 10.0.0.1 50117 Texas 10.0.0.30 80 HTTP UDP 5351 +2010-12-01 23:11:09 Atlanta 10.0.0.1 64789 Texas 10.0.0.30 80 HTTP UDP 2172 +2010-12-01 23:11:10 Atlanta 10.0.0.1 53470 Texas 10.0.0.25 80 HTTP UDP 5455 +2010-12-01 23:11:11 Atlanta 10.0.0.1 64810 Texas 10.0.0.30 80 HTTP UDP 2363 +2010-12-01 23:11:12 Atlanta 10.0.0.1 49457 Texas 10.0.0.25 80 HTTP UDP 3936 +2010-12-01 23:11:13 Atlanta 10.0.0.1 63256 Texas 10.0.0.30 80 HTTP UDP 3512 +2010-12-01 23:11:14 Atlanta 10.0.0.1 64839 Texas 10.0.0.26 80 HTTP UDP 2294 +2010-12-01 23:11:15 Atlanta 10.0.0.1 64571 Texas 10.0.0.26 80 HTTP UDP 4279 +2010-12-01 23:11:16 Atlanta 10.0.0.1 59006 Texas 10.0.0.29 80 HTTP UDP 5937 +2010-12-01 23:11:17 Atlanta 10.0.0.1 59609 Texas 10.0.0.28 80 HTTP UDP 3183 +2010-12-01 23:11:18 Atlanta 10.0.0.1 56877 Texas 10.0.0.26 80 HTTP UDP 5653 +2010-12-01 23:11:19 Atlanta 10.0.0.1 52395 Texas 10.0.0.30 80 HTTP UDP 2480 +2010-12-01 23:11:20 Atlanta 10.0.0.1 64755 Texas 10.0.0.25 80 HTTP UDP 3686 +2010-12-01 23:11:21 Atlanta 10.0.0.1 55949 Texas 10.0.0.26 80 HTTP UDP 4886 +2010-12-01 23:11:22 Atlanta 10.0.0.1 49203 Texas 10.0.0.30 80 HTTP UDP 5906 +2010-12-01 23:11:23 Atlanta 10.0.0.1 49863 Texas 10.0.0.28 80 HTTP UDP 5211 +2010-12-01 23:11:24 Atlanta 10.0.0.1 50930 Texas 10.0.0.26 80 HTTP UDP 2154 +2010-12-01 23:11:25 Atlanta 10.0.0.1 62807 Texas 10.0.0.28 80 HTTP UDP 5946 +2010-12-01 23:11:26 Atlanta 10.0.0.1 63885 Texas 10.0.0.25 80 HTTP UDP 4414 +2010-12-01 23:11:27 Atlanta 10.0.0.1 54106 Texas 10.0.0.25 80 HTTP UDP 5951 +2010-12-01 23:11:28 Atlanta 10.0.0.1 49255 Texas 10.0.0.28 80 HTTP UDP 3987 +2010-12-01 23:11:29 Atlanta 10.0.0.1 61959 Texas 10.0.0.27 80 HTTP UDP 2595 +2010-12-01 23:11:30 Atlanta 10.0.0.1 64122 Texas 10.0.0.26 80 HTTP UDP 4415 +2010-12-01 23:11:31 Atlanta 10.0.0.1 57657 Texas 10.0.0.29 80 HTTP UDP 3347 +2010-12-01 23:11:32 Atlanta 10.0.0.1 49617 Texas 10.0.0.28 80 HTTP UDP 4627 +2010-12-01 23:11:33 Atlanta 10.0.0.1 60442 Texas 10.0.0.30 80 HTTP UDP 3016 +2010-12-01 23:11:34 Atlanta 10.0.0.1 49966 Texas 10.0.0.25 80 HTTP UDP 5072 +2010-12-01 23:11:35 Atlanta 10.0.0.1 53944 Texas 10.0.0.25 80 HTTP UDP 4757 +2010-12-01 23:11:36 Atlanta 10.0.0.1 49757 Texas 10.0.0.25 80 HTTP UDP 4945 +2010-12-01 23:11:37 Atlanta 10.0.0.1 64343 Texas 10.0.0.28 80 HTTP UDP 4365 +2010-12-01 23:11:38 Atlanta 10.0.0.1 52081 Texas 10.0.0.26 80 HTTP UDP 3398 +2010-12-01 23:11:39 Atlanta 10.0.0.1 52421 Texas 10.0.0.29 80 HTTP UDP 2484 +2010-12-01 23:11:40 Atlanta 10.0.0.1 54159 Texas 10.0.0.27 80 HTTP UDP 3562 +2010-12-01 23:11:41 Atlanta 10.0.0.1 59398 Texas 10.0.0.29 80 HTTP UDP 2059 +2010-12-01 23:11:42 Atlanta 10.0.0.1 52343 Texas 10.0.0.30 80 HTTP UDP 3189 +2010-12-01 23:11:43 Atlanta 10.0.0.1 59253 Texas 10.0.0.26 80 HTTP UDP 5223 +2010-12-01 23:11:44 Atlanta 10.0.0.1 60935 Texas 10.0.0.28 80 HTTP UDP 5289 +2010-12-01 23:11:45 Atlanta 10.0.0.1 62240 Texas 10.0.0.28 80 HTTP UDP 4566 +2010-12-01 23:11:46 Atlanta 10.0.0.1 63876 Texas 10.0.0.30 80 HTTP UDP 2595 +2010-12-01 23:11:47 Atlanta 10.0.0.1 56676 Texas 10.0.0.28 80 HTTP UDP 3017 +2010-12-01 23:11:48 Atlanta 10.0.0.1 55922 Texas 10.0.0.30 80 HTTP UDP 2694 +2010-12-01 23:11:49 Atlanta 10.0.0.1 57097 Texas 10.0.0.26 80 HTTP UDP 3017 +2010-12-01 23:11:50 Atlanta 10.0.0.1 49533 Texas 10.0.0.30 80 HTTP UDP 3016 +2010-12-01 23:11:51 Atlanta 10.0.0.1 57465 Texas 10.0.0.29 80 HTTP UDP 5327 +2010-12-01 23:11:52 Atlanta 10.0.0.1 51554 Texas 10.0.0.27 80 HTTP UDP 3936 +2010-12-01 23:11:53 Atlanta 10.0.0.1 58580 Texas 10.0.0.30 80 HTTP UDP 4088 +2010-12-01 23:11:54 Atlanta 10.0.0.1 54577 Texas 10.0.0.25 80 HTTP UDP 2611 +2010-12-01 23:11:55 Atlanta 10.0.0.1 55582 Texas 10.0.0.29 80 HTTP UDP 3469 +2010-12-01 23:11:56 Atlanta 10.0.0.1 58784 Texas 10.0.0.28 80 HTTP UDP 5946 +2010-12-01 23:11:57 Atlanta 10.0.0.1 64075 Texas 10.0.0.28 80 HTTP UDP 5355 +2010-12-01 23:11:58 Atlanta 10.0.0.1 50730 Texas 10.0.0.26 80 HTTP UDP 2432 +2010-12-01 23:11:59 Atlanta 10.0.0.1 64147 Texas 10.0.0.28 80 HTTP UDP 2578 +2010-12-01 23:12:00 Atlanta 10.0.0.1 51801 Texas 10.0.0.27 80 HTTP UDP 5562 +2010-12-01 23:12:01 Atlanta 10.0.0.1 52970 Texas 10.0.0.29 80 HTTP UDP 5348 +2010-12-01 23:12:02 Atlanta 10.0.0.1 51916 Texas 10.0.0.27 80 HTTP UDP 4735 +2010-12-01 23:12:03 Atlanta 10.0.0.1 60167 Texas 10.0.0.30 80 HTTP UDP 5166 +2010-12-01 23:12:04 Atlanta 10.0.0.1 61088 Texas 10.0.0.28 80 HTTP UDP 5197 +2010-12-01 23:12:05 Atlanta 10.0.0.1 56290 Texas 10.0.0.27 80 HTTP UDP 3398 +2010-12-01 23:12:06 Atlanta 10.0.0.1 54399 Texas 10.0.0.26 80 HTTP UDP 4369 +2010-12-01 23:12:07 Atlanta 10.0.0.1 55036 Texas 10.0.0.28 80 HTTP UDP 5436 +2010-12-01 23:12:08 Atlanta 10.0.0.1 57969 Texas 10.0.0.29 80 HTTP UDP 4189 +2010-12-01 23:12:09 Atlanta 10.0.0.1 61773 Texas 10.0.0.26 80 HTTP UDP 5231 +2010-12-01 23:12:10 Atlanta 10.0.0.1 55098 Texas 10.0.0.30 80 HTTP UDP 3292 +2010-12-01 23:12:11 Atlanta 10.0.0.1 51570 Texas 10.0.0.25 80 HTTP UDP 5662 +2010-12-01 23:12:12 Atlanta 10.0.0.1 62360 Texas 10.0.0.25 80 HTTP UDP 4416 +2010-12-01 23:12:13 Atlanta 10.0.0.1 57164 Texas 10.0.0.25 80 HTTP UDP 5662 +2010-12-01 23:12:14 Atlanta 10.0.0.1 58741 Texas 10.0.0.29 80 HTTP UDP 4977 +2010-12-01 23:12:15 Atlanta 10.0.0.1 58122 Texas 10.0.0.30 80 HTTP UDP 5538 +2010-12-01 23:12:16 Atlanta 10.0.0.1 56212 Texas 10.0.0.30 80 HTTP UDP 2551 +2010-12-01 23:12:17 Atlanta 10.0.0.1 56097 Texas 10.0.0.25 80 HTTP UDP 2082 +2010-12-01 23:12:18 Atlanta 10.0.0.1 59416 Texas 10.0.0.25 80 HTTP UDP 5533 +2010-12-01 23:12:19 Atlanta 10.0.0.1 53295 Texas 10.0.0.25 80 HTTP UDP 2391 +2010-12-01 23:12:20 Atlanta 10.0.0.1 57986 Texas 10.0.0.26 80 HTTP UDP 3136 +2010-12-01 23:12:21 Atlanta 10.0.0.1 55654 Texas 10.0.0.27 80 HTTP UDP 5321 +2010-12-01 23:12:22 Atlanta 10.0.0.1 53084 Texas 10.0.0.27 80 HTTP UDP 5243 +2010-12-01 23:12:23 Atlanta 10.0.0.1 54497 Texas 10.0.0.28 80 HTTP UDP 3131 +2010-12-01 23:12:24 Atlanta 10.0.0.1 52212 Texas 10.0.0.25 80 HTTP UDP 3306 +2010-12-01 23:12:25 Atlanta 10.0.0.1 64890 Texas 10.0.0.29 80 HTTP UDP 5371 +2010-12-01 23:12:26 Atlanta 10.0.0.1 63785 Texas 10.0.0.26 80 HTTP UDP 2318 +2010-12-01 23:12:27 Atlanta 10.0.0.1 56131 Texas 10.0.0.25 80 HTTP UDP 3723 +2010-12-01 23:12:28 Atlanta 10.0.0.1 62265 Texas 10.0.0.29 80 HTTP UDP 4193 +2010-12-01 23:12:29 Atlanta 10.0.0.1 51439 Texas 10.0.0.28 80 HTTP UDP 4212 +2010-12-01 23:12:30 Atlanta 10.0.0.1 60273 Texas 10.0.0.25 80 HTTP UDP 5594 +2010-12-01 23:12:31 Atlanta 10.0.0.1 64471 Texas 10.0.0.29 80 HTTP UDP 3122 +2010-12-01 23:12:32 Atlanta 10.0.0.1 56487 Texas 10.0.0.25 80 HTTP UDP 4061 +2010-12-01 23:12:33 Atlanta 10.0.0.1 62756 Texas 10.0.0.28 80 HTTP UDP 4260 +2010-12-01 23:12:34 Atlanta 10.0.0.1 56204 Texas 10.0.0.28 80 HTTP UDP 3256 +2010-12-01 23:12:35 Atlanta 10.0.0.1 61252 Texas 10.0.0.29 80 HTTP UDP 3793 +2010-12-01 23:12:36 Atlanta 10.0.0.1 65043 Texas 10.0.0.25 80 HTTP UDP 5321 +2010-12-01 23:12:37 Atlanta 10.0.0.1 50194 Texas 10.0.0.25 80 HTTP UDP 3478 +2010-12-01 23:12:38 Atlanta 10.0.0.1 59134 Texas 10.0.0.26 80 HTTP UDP 5082 +2010-12-01 23:12:39 Atlanta 10.0.0.1 49490 Texas 10.0.0.26 80 HTTP UDP 5484 +2010-12-01 23:12:40 Atlanta 10.0.0.1 50586 Texas 10.0.0.29 80 HTTP UDP 5501 +2010-12-01 23:12:41 Atlanta 10.0.0.1 64576 Texas 10.0.0.28 80 HTTP UDP 4393 +2010-12-01 23:12:42 Atlanta 10.0.0.1 62166 Texas 10.0.0.29 80 HTTP UDP 4955 +2010-12-01 23:12:43 Atlanta 10.0.0.1 61754 Texas 10.0.0.25 80 HTTP UDP 4506 +2010-12-01 23:12:44 Atlanta 10.0.0.1 64507 Texas 10.0.0.30 80 HTTP UDP 4414 +2010-12-01 23:12:45 Atlanta 10.0.0.1 60815 Texas 10.0.0.26 80 HTTP UDP 4195 +2010-12-01 23:12:46 Atlanta 10.0.0.1 63445 Texas 10.0.0.28 80 HTTP UDP 2519 +2010-12-01 23:12:47 Atlanta 10.0.0.1 64634 Texas 10.0.0.25 80 HTTP UDP 2295 +2010-12-01 23:12:48 Atlanta 10.0.0.1 65166 Texas 10.0.0.28 80 HTTP UDP 3758 +2010-12-01 23:12:49 Atlanta 10.0.0.1 64384 Texas 10.0.0.28 80 HTTP UDP 5347 +2010-12-01 23:12:50 Atlanta 10.0.0.1 50894 Texas 10.0.0.29 80 HTTP UDP 2663 +2010-12-01 23:12:51 Atlanta 10.0.0.1 60399 Texas 10.0.0.25 80 HTTP UDP 3893 +2010-12-01 23:12:52 Atlanta 10.0.0.1 62177 Texas 10.0.0.27 80 HTTP UDP 5327 +2010-12-01 23:12:53 Atlanta 10.0.0.1 65442 Texas 10.0.0.28 80 HTTP UDP 3469 +2010-12-01 23:12:54 Atlanta 10.0.0.1 64962 Texas 10.0.0.25 80 HTTP UDP 4567 +2010-12-01 23:12:55 Atlanta 10.0.0.1 58187 Texas 10.0.0.28 80 HTTP UDP 5231 +2010-12-01 23:12:56 Atlanta 10.0.0.1 64007 Texas 10.0.0.26 80 HTTP UDP 2192 +2010-12-01 23:12:57 Atlanta 10.0.0.1 62664 Texas 10.0.0.28 80 HTTP UDP 4955 +2010-12-01 23:12:58 Atlanta 10.0.0.1 54966 Texas 10.0.0.27 80 HTTP UDP 3872 +2010-12-01 23:12:59 Atlanta 10.0.0.1 62231 Texas 10.0.0.26 80 HTTP UDP 5884 +2010-12-01 23:13:00 Atlanta 10.0.0.1 52508 Texas 10.0.0.30 80 HTTP UDP 3657 +2010-12-01 23:13:01 Atlanta 10.0.0.1 64757 Texas 10.0.0.26 80 HTTP UDP 2946 +2010-12-01 23:13:02 Atlanta 10.0.0.1 55529 Texas 10.0.0.28 80 HTTP UDP 5289 +2010-12-01 23:13:03 Atlanta 10.0.0.1 54215 Texas 10.0.0.29 80 HTTP UDP 3669 +2010-12-01 23:13:04 Atlanta 10.0.0.1 54687 Texas 10.0.0.29 80 HTTP UDP 3935 +2010-12-01 23:13:05 Atlanta 10.0.0.1 57328 Texas 10.0.0.29 80 HTTP UDP 5041 +2010-12-01 23:13:06 Atlanta 10.0.0.1 64076 Texas 10.0.0.28 80 HTTP UDP 3267 +2010-12-01 23:13:07 Atlanta 10.0.0.1 53231 Texas 10.0.0.30 80 HTTP UDP 4530 +2010-12-01 23:13:08 Atlanta 10.0.0.1 62697 Texas 10.0.0.30 80 HTTP UDP 4165 +2010-12-01 23:13:09 Atlanta 10.0.0.1 61758 Texas 10.0.0.25 80 HTTP UDP 2813 +2010-12-01 23:13:10 Atlanta 10.0.0.1 64384 Texas 10.0.0.29 80 HTTP UDP 4160 +2010-12-01 23:13:11 Atlanta 10.0.0.1 63151 Texas 10.0.0.25 80 HTTP UDP 2882 +2010-12-01 23:13:12 Atlanta 10.0.0.1 52198 Texas 10.0.0.30 80 HTTP UDP 4369 +2010-12-01 23:13:13 Atlanta 10.0.0.1 59864 Texas 10.0.0.25 80 HTTP UDP 2273 +2010-12-01 23:13:14 Atlanta 10.0.0.1 53353 Texas 10.0.0.25 80 HTTP UDP 2295 +2010-12-01 23:13:15 Atlanta 10.0.0.1 49265 Texas 10.0.0.27 80 HTTP UDP 2732 +2010-12-01 23:13:16 Atlanta 10.0.0.1 59758 Texas 10.0.0.28 80 HTTP UDP 5628 +2010-12-01 23:13:17 Atlanta 10.0.0.1 55632 Texas 10.0.0.25 80 HTTP UDP 3182 +2010-12-01 23:13:18 Atlanta 10.0.0.1 60574 Texas 10.0.0.25 80 HTTP UDP 4948 +2010-12-01 23:13:19 Atlanta 10.0.0.1 64586 Texas 10.0.0.30 80 HTTP UDP 4293 +2010-12-01 23:13:20 Atlanta 10.0.0.1 58903 Texas 10.0.0.30 80 HTTP UDP 2563 +2010-12-01 23:13:21 Atlanta 10.0.0.1 54989 Texas 10.0.0.27 80 HTTP UDP 5194 +2010-12-01 23:13:22 Atlanta 10.0.0.1 49174 Texas 10.0.0.27 80 HTTP UDP 4316 +2010-12-01 23:13:23 Atlanta 10.0.0.1 62541 Texas 10.0.0.28 80 HTTP UDP 2595 +2010-12-01 23:13:24 Atlanta 10.0.0.1 59729 Texas 10.0.0.25 80 HTTP UDP 3153 +2010-12-01 23:13:25 Atlanta 10.0.0.1 55995 Texas 10.0.0.29 80 HTTP UDP 5747 +2010-12-01 23:13:26 Atlanta 10.0.0.1 55734 Texas 10.0.0.30 80 HTTP UDP 2590 +2010-12-01 23:13:27 Atlanta 10.0.0.1 52059 Texas 10.0.0.28 80 HTTP UDP 3908 +2010-12-01 23:13:28 Atlanta 10.0.0.1 61210 Texas 10.0.0.29 80 HTTP UDP 3104 +2010-12-01 23:13:29 Atlanta 10.0.0.1 50271 Texas 10.0.0.28 80 HTTP UDP 5472 +2010-12-01 23:13:30 Atlanta 10.0.0.1 59220 Texas 10.0.0.30 80 HTTP UDP 2186 +2010-12-01 23:13:31 Atlanta 10.0.0.1 49730 Texas 10.0.0.26 80 HTTP UDP 5044 +2010-12-01 23:13:32 Atlanta 10.0.0.1 51452 Texas 10.0.0.28 80 HTTP UDP 5414 +2010-12-01 23:13:33 Atlanta 10.0.0.1 56572 Texas 10.0.0.27 80 HTTP UDP 2235 +2010-12-01 23:13:34 Atlanta 10.0.0.1 49294 Texas 10.0.0.26 80 HTTP UDP 5285 +2010-12-01 23:13:35 Atlanta 10.0.0.1 49830 Texas 10.0.0.25 80 HTTP UDP 2067 +2010-12-01 23:13:36 Atlanta 10.0.0.1 53678 Texas 10.0.0.27 80 HTTP UDP 3398 +2010-12-01 23:13:37 Atlanta 10.0.0.1 57800 Texas 10.0.0.27 80 HTTP UDP 5951 +2010-12-01 23:13:38 Atlanta 10.0.0.1 64945 Texas 10.0.0.26 80 HTTP UDP 2263 +2010-12-01 23:13:39 Atlanta 10.0.0.1 54650 Texas 10.0.0.30 80 HTTP UDP 2708 +2010-12-01 23:13:40 Atlanta 10.0.0.1 57249 Texas 10.0.0.25 80 HTTP UDP 4312 +2010-12-01 23:13:41 Atlanta 10.0.0.1 50259 Texas 10.0.0.27 80 HTTP UDP 2022 +2010-12-01 23:13:42 Atlanta 10.0.0.1 50206 Texas 10.0.0.30 80 HTTP UDP 4414 +2010-12-01 23:13:43 Atlanta 10.0.0.1 62080 Texas 10.0.0.28 80 HTTP UDP 4407 +2010-12-01 23:13:44 Atlanta 10.0.0.1 49660 Texas 10.0.0.27 80 HTTP UDP 2366 +2010-12-01 23:13:45 Atlanta 10.0.0.1 57962 Texas 10.0.0.25 80 HTTP UDP 2790 +2010-12-01 23:13:46 Atlanta 10.0.0.1 61751 Texas 10.0.0.28 80 HTTP UDP 2734 +2010-12-01 23:13:47 Atlanta 10.0.0.1 57170 Texas 10.0.0.28 80 HTTP UDP 5856 +2010-12-01 23:13:48 Atlanta 10.0.0.1 54160 Texas 10.0.0.26 80 HTTP UDP 4211 +2010-12-01 23:13:49 Atlanta 10.0.0.1 63362 Texas 10.0.0.27 80 HTTP UDP 2050 +2010-12-01 23:13:50 Atlanta 10.0.0.1 53863 Texas 10.0.0.29 80 HTTP UDP 2432 +2010-12-01 23:13:51 Atlanta 10.0.0.1 57450 Texas 10.0.0.26 80 HTTP UDP 5041 +2010-12-01 23:13:52 Atlanta 10.0.0.1 64382 Texas 10.0.0.25 80 HTTP UDP 4176 +2010-12-01 23:13:53 Atlanta 10.0.0.1 55565 Texas 10.0.0.30 80 HTTP UDP 4826 +2010-12-01 23:13:54 Atlanta 10.0.0.1 61199 Texas 10.0.0.29 80 HTTP UDP 4400 +2010-12-01 23:13:55 Atlanta 10.0.0.1 61381 Texas 10.0.0.27 80 HTTP UDP 3589 +2010-12-01 23:13:56 Atlanta 10.0.0.1 57526 Texas 10.0.0.30 80 HTTP UDP 5082 +2010-12-01 23:13:57 Atlanta 10.0.0.1 56669 Texas 10.0.0.25 80 HTTP UDP 5997 +2010-12-01 23:13:58 Atlanta 10.0.0.1 54252 Texas 10.0.0.30 80 HTTP UDP 5422 +2010-12-01 23:13:59 Atlanta 10.0.0.1 57388 Texas 10.0.0.28 80 HTTP UDP 3959 +2010-12-01 23:14:00 Atlanta 10.0.0.1 50586 Texas 10.0.0.26 80 HTTP UDP 3621 +2010-12-01 23:14:01 Atlanta 10.0.0.1 57378 Texas 10.0.0.27 80 HTTP UDP 2079 +2010-12-01 23:14:02 Atlanta 10.0.0.1 56088 Texas 10.0.0.26 80 HTTP UDP 4936 +2010-12-01 23:14:03 Atlanta 10.0.0.1 53445 Texas 10.0.0.28 80 HTTP UDP 5256 +2010-12-01 23:14:04 Atlanta 10.0.0.1 54803 Texas 10.0.0.27 80 HTTP UDP 5710 +2010-12-01 23:14:05 Atlanta 10.0.0.1 63210 Texas 10.0.0.30 80 HTTP UDP 2818 +2010-12-01 23:14:06 Atlanta 10.0.0.1 49594 Texas 10.0.0.27 80 HTTP UDP 2532 +2010-12-01 23:14:07 Atlanta 10.0.0.1 59015 Texas 10.0.0.27 80 HTTP UDP 5340 +2010-12-01 23:14:08 Atlanta 10.0.0.1 64134 Texas 10.0.0.30 80 HTTP UDP 5775 +2010-12-01 23:14:09 Atlanta 10.0.0.1 55972 Texas 10.0.0.26 80 HTTP UDP 2643 +2010-12-01 23:14:10 Atlanta 10.0.0.1 54789 Texas 10.0.0.28 80 HTTP UDP 4281 +2010-12-01 23:14:11 Atlanta 10.0.0.1 53564 Texas 10.0.0.25 80 HTTP UDP 4899 +2010-12-01 23:14:12 Atlanta 10.0.0.1 54628 Texas 10.0.0.25 80 HTTP UDP 4794 +2010-12-01 23:14:13 Atlanta 10.0.0.1 62397 Texas 10.0.0.27 80 HTTP UDP 3601 +2010-12-01 23:14:14 Atlanta 10.0.0.1 63874 Texas 10.0.0.29 80 HTTP UDP 4093 +2010-12-01 23:14:15 Atlanta 10.0.0.1 49466 Texas 10.0.0.26 80 HTTP UDP 5435 +2010-12-01 23:14:16 Atlanta 10.0.0.1 57482 Texas 10.0.0.28 80 HTTP UDP 4777 +2010-12-01 23:14:17 Atlanta 10.0.0.1 58145 Texas 10.0.0.29 80 HTTP UDP 4349 +2010-12-01 23:14:18 Atlanta 10.0.0.1 59966 Texas 10.0.0.25 80 HTTP UDP 4316 +2010-12-01 23:14:19 Atlanta 10.0.0.1 60462 Texas 10.0.0.27 80 HTTP UDP 2403 +2010-12-01 23:14:20 Atlanta 10.0.0.1 61371 Texas 10.0.0.29 80 HTTP UDP 3934 +2010-12-01 23:14:21 Atlanta 10.0.0.1 62870 Texas 10.0.0.30 80 HTTP UDP 5763 +2010-12-01 23:14:22 Atlanta 10.0.0.1 65158 Texas 10.0.0.25 80 HTTP UDP 3318 +2010-12-01 23:14:23 Atlanta 10.0.0.1 64491 Texas 10.0.0.27 80 HTTP UDP 4902 +2010-12-01 23:14:24 Atlanta 10.0.0.1 59894 Texas 10.0.0.25 80 HTTP UDP 4043 +2010-12-01 23:14:25 Atlanta 10.0.0.1 54682 Texas 10.0.0.26 80 HTTP UDP 4416 +2010-12-01 23:14:26 Atlanta 10.0.0.1 53804 Texas 10.0.0.26 80 HTTP UDP 4058 +2010-12-01 23:14:27 Atlanta 10.0.0.1 58643 Texas 10.0.0.30 80 HTTP UDP 2656 +2010-12-01 23:14:28 Atlanta 10.0.0.1 61083 Texas 10.0.0.27 80 HTTP UDP 3736 +2010-12-01 23:14:29 Atlanta 10.0.0.1 53437 Texas 10.0.0.28 80 HTTP UDP 2881 +2010-12-01 23:14:30 Atlanta 10.0.0.1 52991 Texas 10.0.0.28 80 HTTP UDP 5951 +2010-12-01 23:14:31 Atlanta 10.0.0.1 52994 Texas 10.0.0.29 80 HTTP UDP 3736 +2010-12-01 23:14:32 Atlanta 10.0.0.1 51577 Texas 10.0.0.26 80 HTTP UDP 3921 +2010-12-01 23:14:33 Atlanta 10.0.0.1 57456 Texas 10.0.0.28 80 HTTP UDP 3934 +2010-12-01 23:14:34 Atlanta 10.0.0.1 56834 Texas 10.0.0.25 80 HTTP UDP 3970 +2010-12-01 23:14:35 Atlanta 10.0.0.1 54102 Texas 10.0.0.29 80 HTTP UDP 5343 +2010-12-01 23:14:36 Atlanta 10.0.0.1 62834 Texas 10.0.0.25 80 HTTP UDP 5511 +2010-12-01 23:14:37 Atlanta 10.0.0.1 51802 Texas 10.0.0.30 80 HTTP UDP 2355 +2010-12-01 23:14:38 Atlanta 10.0.0.1 57080 Texas 10.0.0.30 80 HTTP UDP 5347 +2010-12-01 23:14:39 Atlanta 10.0.0.1 63707 Texas 10.0.0.29 80 HTTP UDP 2804 +2010-12-01 23:14:40 Atlanta 10.0.0.1 65489 Texas 10.0.0.30 80 HTTP UDP 4959 +2010-12-01 23:14:41 Atlanta 10.0.0.1 51177 Texas 10.0.0.27 80 HTTP UDP 2189 +2010-12-01 23:14:42 Atlanta 10.0.0.1 58555 Texas 10.0.0.29 80 HTTP UDP 5370 +2010-12-01 23:14:43 Atlanta 10.0.0.1 60436 Texas 10.0.0.25 80 HTTP UDP 4651 +2010-12-01 23:14:44 Atlanta 10.0.0.1 53481 Texas 10.0.0.29 80 HTTP UDP 4319 +2010-12-01 23:14:45 Atlanta 10.0.0.1 57220 Texas 10.0.0.28 80 HTTP UDP 3079 +2010-12-01 23:14:46 Atlanta 10.0.0.1 56003 Texas 10.0.0.27 80 HTTP UDP 2391 +2010-12-01 23:14:47 Atlanta 10.0.0.1 50319 Texas 10.0.0.29 80 HTTP UDP 4717 +2010-12-01 23:14:48 Atlanta 10.0.0.1 49923 Texas 10.0.0.28 80 HTTP UDP 4962 +2010-12-01 23:14:49 Atlanta 10.0.0.1 53886 Texas 10.0.0.30 80 HTTP UDP 3733 +2010-12-01 23:14:50 Atlanta 10.0.0.1 50334 Texas 10.0.0.30 80 HTTP UDP 5501 +2010-12-01 23:14:51 Atlanta 10.0.0.1 58751 Texas 10.0.0.25 80 HTTP UDP 4855 +2010-12-01 23:14:52 Atlanta 10.0.0.1 60728 Texas 10.0.0.27 80 HTTP UDP 4212 +2010-12-01 23:14:53 Atlanta 10.0.0.1 63623 Texas 10.0.0.26 80 HTTP UDP 3864 +2010-12-01 23:14:54 Atlanta 10.0.0.1 58449 Texas 10.0.0.26 80 HTTP UDP 4567 +2010-12-01 23:14:55 Atlanta 10.0.0.1 62935 Texas 10.0.0.28 80 HTTP UDP 5366 +2010-12-01 23:14:56 Atlanta 10.0.0.1 61436 Texas 10.0.0.26 80 HTTP UDP 3032 +2010-12-01 23:14:57 Atlanta 10.0.0.1 55547 Texas 10.0.0.29 80 HTTP UDP 5002 +2010-12-01 23:14:58 Atlanta 10.0.0.1 59872 Texas 10.0.0.29 80 HTTP UDP 4893 +2010-12-01 23:14:59 Atlanta 10.0.0.1 49540 Texas 10.0.0.26 80 HTTP UDP 4006 +2010-12-01 23:15:00 Atlanta 10.0.0.1 54919 Texas 10.0.0.28 80 HTTP UDP 2015 +2010-12-01 23:15:01 Atlanta 10.0.0.1 62714 Texas 10.0.0.27 80 HTTP UDP 3009 +2010-12-01 23:15:02 Atlanta 10.0.0.1 54313 Texas 10.0.0.30 80 HTTP UDP 2924 +2010-12-01 23:15:03 Atlanta 10.0.0.1 62900 Texas 10.0.0.27 80 HTTP UDP 3609 +2010-12-01 23:15:04 Atlanta 10.0.0.1 54110 Texas 10.0.0.26 80 HTTP UDP 2651 +2010-12-01 23:15:05 Atlanta 10.0.0.1 58036 Texas 10.0.0.25 80 HTTP UDP 4395 +2010-12-01 23:15:06 Atlanta 10.0.0.1 63654 Texas 10.0.0.26 80 HTTP UDP 2979 +2010-12-01 23:15:07 Atlanta 10.0.0.1 59436 Texas 10.0.0.29 80 HTTP UDP 3995 +2010-12-01 23:15:08 Atlanta 10.0.0.1 63499 Texas 10.0.0.26 80 HTTP UDP 5946 +2010-12-01 23:15:09 Atlanta 10.0.0.1 52083 Texas 10.0.0.28 80 HTTP UDP 3618 +2010-12-01 23:15:10 Atlanta 10.0.0.1 59769 Texas 10.0.0.27 80 HTTP UDP 2788 +2010-12-01 23:15:11 Atlanta 10.0.0.1 58515 Texas 10.0.0.25 80 HTTP UDP 2561 +2010-12-01 23:15:12 Atlanta 10.0.0.1 57119 Texas 10.0.0.27 80 HTTP UDP 3766 +2010-12-01 23:15:13 Atlanta 10.0.0.1 58387 Texas 10.0.0.30 80 HTTP UDP 2611 +2010-12-01 23:15:14 Atlanta 10.0.0.1 57940 Texas 10.0.0.25 80 HTTP UDP 4164 +2010-12-01 23:15:15 Atlanta 10.0.0.1 59636 Texas 10.0.0.26 80 HTTP UDP 2215 +2010-12-01 23:15:16 Atlanta 10.0.0.1 52318 Texas 10.0.0.29 80 HTTP UDP 3511 +2010-12-01 23:15:17 Atlanta 10.0.0.1 60988 Texas 10.0.0.30 80 HTTP UDP 5033 +2010-12-01 23:15:18 Atlanta 10.0.0.1 49399 Texas 10.0.0.29 80 HTTP UDP 4399 +2010-12-01 23:15:19 Atlanta 10.0.0.1 58205 Texas 10.0.0.29 80 HTTP UDP 5468 +2010-12-01 23:15:20 Atlanta 10.0.0.1 65105 Texas 10.0.0.26 80 HTTP UDP 4103 +2010-12-01 23:15:21 Atlanta 10.0.0.1 62336 Texas 10.0.0.25 80 HTTP UDP 5963 +2010-12-01 23:15:22 Atlanta 10.0.0.1 57164 Texas 10.0.0.25 80 HTTP UDP 2595 +2010-12-01 23:15:23 Atlanta 10.0.0.1 64695 Texas 10.0.0.28 80 HTTP UDP 3765 +2010-12-01 23:15:24 Atlanta 10.0.0.1 63395 Texas 10.0.0.25 80 HTTP UDP 4644 +2010-12-01 23:15:25 Atlanta 10.0.0.1 51381 Texas 10.0.0.29 80 HTTP UDP 3708 +2010-12-01 23:15:26 Atlanta 10.0.0.1 62772 Texas 10.0.0.27 80 HTTP UDP 4506 +2010-12-01 23:15:27 Atlanta 10.0.0.1 61720 Texas 10.0.0.25 80 HTTP UDP 5047 +2010-12-01 23:15:28 Atlanta 10.0.0.1 60699 Texas 10.0.0.28 80 HTTP UDP 4088 +2010-12-01 23:15:29 Atlanta 10.0.0.1 62879 Texas 10.0.0.27 80 HTTP UDP 3052 +2010-12-01 23:15:30 Atlanta 10.0.0.1 56995 Texas 10.0.0.29 80 HTTP UDP 5421 +2010-12-01 23:15:31 Atlanta 10.0.0.1 54681 Texas 10.0.0.25 80 HTTP UDP 5321 +2010-12-01 23:15:32 Atlanta 10.0.0.1 58418 Texas 10.0.0.29 80 HTTP UDP 3857 +2010-12-01 23:15:33 Atlanta 10.0.0.1 63507 Texas 10.0.0.27 80 HTTP UDP 3657 +2010-12-01 23:15:34 Atlanta 10.0.0.1 57674 Texas 10.0.0.25 80 HTTP UDP 3460 +2010-12-01 23:15:35 Atlanta 10.0.0.1 63754 Texas 10.0.0.26 80 HTTP UDP 4867 +2010-12-01 23:15:36 Atlanta 10.0.0.1 61509 Texas 10.0.0.27 80 HTTP UDP 5290 +2010-12-01 23:15:37 Atlanta 10.0.0.1 53729 Texas 10.0.0.29 80 HTTP UDP 2652 +2010-12-01 23:15:38 Atlanta 10.0.0.1 52749 Texas 10.0.0.25 80 HTTP UDP 4093 +2010-12-01 23:15:39 Atlanta 10.0.0.1 57026 Texas 10.0.0.25 80 HTTP UDP 4400 +2010-12-01 23:15:40 Atlanta 10.0.0.1 51741 Texas 10.0.0.29 80 HTTP UDP 3345 +2010-12-01 23:15:41 Atlanta 10.0.0.1 61728 Texas 10.0.0.29 80 HTTP UDP 4437 +2010-12-01 23:15:42 Atlanta 10.0.0.1 55699 Texas 10.0.0.27 80 HTTP UDP 2578 +2010-12-01 23:15:43 Atlanta 10.0.0.1 56072 Texas 10.0.0.29 80 HTTP UDP 4903 +2010-12-01 23:15:44 Atlanta 10.0.0.1 65125 Texas 10.0.0.27 80 HTTP UDP 2559 +2010-12-01 23:15:45 Atlanta 10.0.0.1 64829 Texas 10.0.0.29 80 HTTP UDP 5817 +2010-12-01 23:15:46 Atlanta 10.0.0.1 61768 Texas 10.0.0.30 80 HTTP UDP 4415 +2010-12-01 23:15:47 Atlanta 10.0.0.1 51946 Texas 10.0.0.25 80 HTTP UDP 5775 +2010-12-01 23:15:48 Atlanta 10.0.0.1 55640 Texas 10.0.0.25 80 HTTP UDP 4744 +2010-12-01 23:15:49 Atlanta 10.0.0.1 62610 Texas 10.0.0.26 80 HTTP UDP 3182 +2010-12-01 23:15:50 Atlanta 10.0.0.1 58696 Texas 10.0.0.26 80 HTTP UDP 4241 +2010-12-01 23:15:51 Atlanta 10.0.0.1 61102 Texas 10.0.0.30 80 HTTP UDP 2643 +2010-12-01 23:15:52 Atlanta 10.0.0.1 51407 Texas 10.0.0.25 80 HTTP UDP 2734 +2010-12-01 23:15:53 Atlanta 10.0.0.1 51256 Texas 10.0.0.28 80 HTTP UDP 2273 +2010-12-01 23:15:54 Atlanta 10.0.0.1 49176 Texas 10.0.0.29 80 HTTP UDP 2022 +2010-12-01 23:15:55 Atlanta 10.0.0.1 53506 Texas 10.0.0.25 80 HTTP UDP 5998 +2010-12-01 23:15:56 Atlanta 10.0.0.1 60921 Texas 10.0.0.29 80 HTTP UDP 5582 +2010-12-01 23:15:57 Atlanta 10.0.0.1 60556 Texas 10.0.0.25 80 HTTP UDP 5119 +2010-12-01 23:15:58 Atlanta 10.0.0.1 58575 Texas 10.0.0.27 80 HTTP UDP 5720 +2010-12-01 23:15:59 Atlanta 10.0.0.1 56122 Texas 10.0.0.26 80 HTTP UDP 5907 +2010-12-01 23:16:00 Atlanta 10.0.0.1 57245 Texas 10.0.0.30 80 HTTP UDP 4293 +2010-12-01 23:16:01 Atlanta 10.0.0.1 58037 Texas 10.0.0.30 80 HTTP UDP 4193 +2010-12-01 23:16:02 Atlanta 10.0.0.1 58692 Texas 10.0.0.30 80 HTTP UDP 2189 +2010-12-01 23:16:03 Atlanta 10.0.0.1 59773 Texas 10.0.0.26 80 HTTP UDP 5946 +2010-12-01 23:16:04 Atlanta 10.0.0.1 51975 Texas 10.0.0.30 80 HTTP UDP 5684 +2010-12-01 23:16:05 Atlanta 10.0.0.1 61657 Texas 10.0.0.29 80 HTTP UDP 5624 +2010-12-01 23:16:06 Atlanta 10.0.0.1 59508 Texas 10.0.0.26 80 HTTP UDP 5673 +2010-12-01 23:16:07 Atlanta 10.0.0.1 50585 Texas 10.0.0.25 80 HTTP UDP 5907 +2010-12-01 23:16:08 Atlanta 10.0.0.1 63404 Texas 10.0.0.27 80 HTTP UDP 5098 +2010-12-01 23:16:09 Atlanta 10.0.0.1 62531 Texas 10.0.0.27 80 HTTP UDP 3146 +2010-12-01 23:16:10 Atlanta 10.0.0.1 59652 Texas 10.0.0.27 80 HTTP UDP 3673 +2010-12-01 23:16:11 Atlanta 10.0.0.1 52131 Texas 10.0.0.27 80 HTTP UDP 4743 +2010-12-01 23:16:12 Atlanta 10.0.0.1 60002 Texas 10.0.0.30 80 HTTP UDP 3306 +2010-12-01 23:16:13 Atlanta 10.0.0.1 62655 Texas 10.0.0.26 80 HTTP UDP 3572 +2010-12-01 23:16:14 Atlanta 10.0.0.1 61233 Texas 10.0.0.28 80 HTTP UDP 5584 +2010-12-01 23:16:15 Atlanta 10.0.0.1 60031 Texas 10.0.0.30 80 HTTP UDP 4349 +2010-12-01 23:16:16 Atlanta 10.0.0.1 58779 Texas 10.0.0.29 80 HTTP UDP 5755 +2010-12-01 23:16:17 Atlanta 10.0.0.1 64958 Texas 10.0.0.30 80 HTTP UDP 2001 +2010-12-01 23:16:18 Atlanta 10.0.0.1 62725 Texas 10.0.0.27 80 HTTP UDP 5952 +2010-12-01 23:16:19 Atlanta 10.0.0.1 55614 Texas 10.0.0.30 80 HTTP UDP 5940 +2010-12-01 23:16:20 Atlanta 10.0.0.1 63604 Texas 10.0.0.25 80 HTTP UDP 5338 +2010-12-01 23:16:21 Atlanta 10.0.0.1 54934 Texas 10.0.0.29 80 HTTP UDP 5033 +2010-12-01 23:16:22 Atlanta 10.0.0.1 63321 Texas 10.0.0.27 80 HTTP UDP 3455 +2010-12-01 23:16:23 Atlanta 10.0.0.1 57106 Texas 10.0.0.29 80 HTTP UDP 3183 +2010-12-01 23:16:24 Atlanta 10.0.0.1 57022 Texas 10.0.0.27 80 HTTP UDP 3318 +2010-12-01 23:16:25 Atlanta 10.0.0.1 53725 Texas 10.0.0.26 80 HTTP UDP 3538 +2010-12-01 23:16:26 Atlanta 10.0.0.1 52532 Texas 10.0.0.25 80 HTTP UDP 5747 +2010-12-01 23:16:27 Atlanta 10.0.0.1 57503 Texas 10.0.0.27 80 HTTP UDP 2988 +2010-12-01 23:16:28 Atlanta 10.0.0.1 59408 Texas 10.0.0.26 80 HTTP UDP 4633 +2010-12-01 23:16:29 Atlanta 10.0.0.1 59867 Texas 10.0.0.27 80 HTTP UDP 2934 +2010-12-01 23:16:30 Atlanta 10.0.0.1 57318 Texas 10.0.0.25 80 HTTP UDP 2569 +2010-12-01 23:16:31 Atlanta 10.0.0.1 50024 Texas 10.0.0.29 80 HTTP UDP 5029 +2010-12-01 23:16:32 Atlanta 10.0.0.1 49474 Texas 10.0.0.27 80 HTTP UDP 3160 +2010-12-01 23:16:33 Atlanta 10.0.0.1 58019 Texas 10.0.0.27 80 HTTP UDP 2945 +2010-12-01 23:16:34 Atlanta 10.0.0.1 53050 Texas 10.0.0.25 80 HTTP UDP 2611 +2010-12-01 23:16:35 Atlanta 10.0.0.1 49347 Texas 10.0.0.30 80 HTTP UDP 2850 +2010-12-01 23:16:36 Atlanta 10.0.0.1 54658 Texas 10.0.0.26 80 HTTP UDP 3114 +2010-12-01 23:16:37 Atlanta 10.0.0.1 62155 Texas 10.0.0.27 80 HTTP UDP 4211 +2010-12-01 23:16:38 Atlanta 10.0.0.1 63877 Texas 10.0.0.25 80 HTTP UDP 5392 +2010-12-01 23:16:39 Atlanta 10.0.0.1 54167 Texas 10.0.0.30 80 HTTP UDP 5327 +2010-12-01 23:16:40 Atlanta 10.0.0.1 49470 Texas 10.0.0.25 80 HTTP UDP 3063 +2010-12-01 23:16:41 Atlanta 10.0.0.1 61976 Texas 10.0.0.28 80 HTTP UDP 4212 +2010-12-01 23:16:42 Atlanta 10.0.0.1 57610 Texas 10.0.0.30 80 HTTP UDP 4260 +2010-12-01 23:16:43 Atlanta 10.0.0.1 59475 Texas 10.0.0.26 80 HTTP UDP 2265 +2010-12-01 23:16:44 Atlanta 10.0.0.1 54767 Texas 10.0.0.27 80 HTTP UDP 2231 +2010-12-01 23:16:45 Atlanta 10.0.0.1 52547 Texas 10.0.0.25 80 HTTP UDP 3135 +2010-12-01 23:16:46 Atlanta 10.0.0.1 58845 Texas 10.0.0.29 80 HTTP UDP 2596 +2010-12-01 23:16:47 Atlanta 10.0.0.1 51280 Texas 10.0.0.25 80 HTTP UDP 4546 +2010-12-01 23:16:48 Atlanta 10.0.0.1 57035 Texas 10.0.0.30 80 HTTP UDP 3861 +2010-12-01 23:16:49 Atlanta 10.0.0.1 61762 Texas 10.0.0.27 80 HTTP UDP 2098 +2010-12-01 23:16:50 Atlanta 10.0.0.1 60270 Texas 10.0.0.27 80 HTTP UDP 4711 +2010-12-01 23:16:51 Atlanta 10.0.0.1 59014 Texas 10.0.0.27 80 HTTP UDP 2693 +2010-12-01 23:16:52 Atlanta 10.0.0.1 63572 Texas 10.0.0.25 80 HTTP UDP 2782 +2010-12-01 23:16:53 Atlanta 10.0.0.1 63089 Texas 10.0.0.28 80 HTTP UDP 5370 +2010-12-01 23:16:54 Atlanta 10.0.0.1 58988 Texas 10.0.0.27 80 HTTP UDP 2571 +2010-12-01 23:16:55 Atlanta 10.0.0.1 64289 Texas 10.0.0.29 80 HTTP UDP 5436 +2010-12-01 23:16:56 Atlanta 10.0.0.1 52009 Texas 10.0.0.25 80 HTTP UDP 4252 +2010-12-01 23:16:57 Atlanta 10.0.0.1 58512 Texas 10.0.0.30 80 HTTP UDP 2311 +2010-12-01 23:16:58 Atlanta 10.0.0.1 54179 Texas 10.0.0.25 80 HTTP UDP 3609 +2010-12-01 23:16:59 Atlanta 10.0.0.1 59831 Texas 10.0.0.25 80 HTTP UDP 5545 +2010-12-01 23:17:00 Atlanta 10.0.0.1 52448 Texas 10.0.0.29 80 HTTP UDP 2479 +2010-12-01 23:17:01 Atlanta 10.0.0.1 63747 Texas 10.0.0.26 80 HTTP UDP 5455 +2010-12-01 23:17:02 Atlanta 10.0.0.1 56531 Texas 10.0.0.28 80 HTTP UDP 3669 +2010-12-01 23:17:03 Atlanta 10.0.0.1 49691 Texas 10.0.0.28 80 HTTP UDP 2523 +2010-12-01 23:17:04 Atlanta 10.0.0.1 53945 Texas 10.0.0.27 80 HTTP UDP 2660 +2010-12-01 23:17:05 Atlanta 10.0.0.1 65170 Texas 10.0.0.25 80 HTTP UDP 4238 +2010-12-01 23:17:06 Atlanta 10.0.0.1 55473 Texas 10.0.0.29 80 HTTP UDP 5309 +2010-12-01 23:17:07 Atlanta 10.0.0.1 59938 Texas 10.0.0.25 80 HTTP UDP 3025 +2010-12-01 23:17:08 Atlanta 10.0.0.1 59646 Texas 10.0.0.30 80 HTTP UDP 3419 +2010-12-01 23:17:09 Atlanta 10.0.0.1 49738 Texas 10.0.0.29 80 HTTP UDP 5668 +2010-12-01 23:17:10 Atlanta 10.0.0.1 64101 Texas 10.0.0.28 80 HTTP UDP 3114 +2010-12-01 23:17:11 Atlanta 10.0.0.1 55523 Texas 10.0.0.27 80 HTTP UDP 2611 +2010-12-01 23:17:12 Atlanta 10.0.0.1 50954 Texas 10.0.0.30 80 HTTP UDP 2007 +2010-12-01 23:17:13 Atlanta 10.0.0.1 64392 Texas 10.0.0.29 80 HTTP UDP 5315 +2010-12-01 23:17:14 Atlanta 10.0.0.1 49940 Texas 10.0.0.26 80 HTTP UDP 4179 +2010-12-01 23:17:15 Atlanta 10.0.0.1 55394 Texas 10.0.0.26 80 HTTP UDP 4379 +2010-12-01 23:17:16 Atlanta 10.0.0.1 60958 Texas 10.0.0.25 80 HTTP UDP 4868 +2010-12-01 23:17:17 Atlanta 10.0.0.1 58247 Texas 10.0.0.26 80 HTTP UDP 5119 +2010-12-01 23:17:18 Atlanta 10.0.0.1 56162 Texas 10.0.0.27 80 HTTP UDP 2587 +2010-12-01 23:17:19 Atlanta 10.0.0.1 52318 Texas 10.0.0.26 80 HTTP UDP 5370 +2010-12-01 23:17:20 Atlanta 10.0.0.1 52563 Texas 10.0.0.30 80 HTTP UDP 3743 +2010-12-01 23:17:21 Atlanta 10.0.0.1 62141 Texas 10.0.0.28 80 HTTP UDP 5946 +2010-12-01 23:17:22 Atlanta 10.0.0.1 60504 Texas 10.0.0.27 80 HTTP UDP 3497 +2010-12-01 23:17:23 Atlanta 10.0.0.1 58542 Texas 10.0.0.30 80 HTTP UDP 4506 +2010-12-01 23:17:24 Atlanta 10.0.0.1 55218 Texas 10.0.0.27 80 HTTP UDP 3875 +2010-12-01 23:17:25 Atlanta 10.0.0.1 57242 Texas 10.0.0.25 80 HTTP UDP 2391 +2010-12-01 23:17:26 Atlanta 10.0.0.1 63052 Texas 10.0.0.29 80 HTTP UDP 4427 +2010-12-01 23:17:27 Atlanta 10.0.0.1 59594 Texas 10.0.0.29 80 HTTP UDP 3861 +2010-12-01 23:17:28 Atlanta 10.0.0.1 54465 Texas 10.0.0.26 80 HTTP UDP 4437 +2010-12-01 23:17:29 Atlanta 10.0.0.1 57414 Texas 10.0.0.30 80 HTTP UDP 5738 +2010-12-01 23:17:30 Atlanta 10.0.0.1 55208 Texas 10.0.0.27 80 HTTP UDP 3696 +2010-12-01 23:17:31 Atlanta 10.0.0.1 60395 Texas 10.0.0.25 80 HTTP UDP 3905 +2010-12-01 23:17:32 Atlanta 10.0.0.1 54517 Texas 10.0.0.30 80 HTTP UDP 5280 +2010-12-01 23:17:33 Atlanta 10.0.0.1 57595 Texas 10.0.0.28 80 HTTP UDP 3630 +2010-12-01 23:17:34 Atlanta 10.0.0.1 58810 Texas 10.0.0.27 80 HTTP UDP 3761 +2010-12-01 23:17:35 Atlanta 10.0.0.1 55435 Texas 10.0.0.28 80 HTTP UDP 2046 +2010-12-01 23:17:36 Atlanta 10.0.0.1 52072 Texas 10.0.0.26 80 HTTP UDP 4006 +2010-12-01 23:17:37 Atlanta 10.0.0.1 52138 Texas 10.0.0.30 80 HTTP UDP 4374 +2010-12-01 23:17:38 Atlanta 10.0.0.1 51802 Texas 10.0.0.25 80 HTTP UDP 5501 +2010-12-01 23:17:39 Atlanta 10.0.0.1 53138 Texas 10.0.0.28 80 HTTP UDP 4399 +2010-12-01 23:17:40 Atlanta 10.0.0.1 52528 Texas 10.0.0.26 80 HTTP UDP 5371 +2010-12-01 23:17:41 Atlanta 10.0.0.1 60851 Texas 10.0.0.27 80 HTTP UDP 4411 +2010-12-01 23:17:42 Atlanta 10.0.0.1 59792 Texas 10.0.0.30 80 HTTP UDP 5019 +2010-12-01 23:17:43 Atlanta 10.0.0.1 55478 Texas 10.0.0.27 80 HTTP UDP 2484 +2010-12-01 23:17:44 Atlanta 10.0.0.1 49216 Texas 10.0.0.27 80 HTTP UDP 2895 +2010-12-01 23:17:45 Atlanta 10.0.0.1 56665 Texas 10.0.0.26 80 HTTP UDP 5533 +2010-12-01 23:17:46 Atlanta 10.0.0.1 62715 Texas 10.0.0.29 80 HTTP UDP 5321 +2010-12-01 23:17:47 Atlanta 10.0.0.1 56059 Texas 10.0.0.26 80 HTTP UDP 5935 +2010-12-01 23:17:48 Atlanta 10.0.0.1 63358 Texas 10.0.0.30 80 HTTP UDP 3905 +2010-12-01 23:17:49 Atlanta 10.0.0.1 58072 Texas 10.0.0.27 80 HTTP UDP 4416 +2010-12-01 23:17:50 Atlanta 10.0.0.1 63685 Texas 10.0.0.25 80 HTTP UDP 3781 +2010-12-01 23:17:51 Atlanta 10.0.0.1 57515 Texas 10.0.0.26 80 HTTP UDP 3628 +2010-12-01 23:17:52 Atlanta 10.0.0.1 57498 Texas 10.0.0.25 80 HTTP UDP 2363 +2010-12-01 23:17:53 Atlanta 10.0.0.1 53471 Texas 10.0.0.25 80 HTTP UDP 5243 +2010-12-01 23:17:54 Atlanta 10.0.0.1 50194 Texas 10.0.0.30 80 HTTP UDP 2294 +2010-12-01 23:17:55 Atlanta 10.0.0.1 49438 Texas 10.0.0.27 80 HTTP UDP 4627 +2010-12-01 23:17:56 Atlanta 10.0.0.1 51193 Texas 10.0.0.29 80 HTTP UDP 3189 +2010-12-01 23:17:57 Atlanta 10.0.0.1 62147 Texas 10.0.0.29 80 HTTP UDP 5309 +2010-12-01 23:17:58 Atlanta 10.0.0.1 65523 Texas 10.0.0.26 80 HTTP UDP 3857 +2010-12-01 23:17:59 Atlanta 10.0.0.1 64473 Texas 10.0.0.29 80 HTTP UDP 3696 +2010-12-01 23:18:00 Atlanta 10.0.0.1 50430 Texas 10.0.0.30 80 HTTP UDP 2881 +2010-12-01 23:18:01 Atlanta 10.0.0.1 49937 Texas 10.0.0.25 80 HTTP UDP 2985 +2010-12-01 23:18:02 Atlanta 10.0.0.1 52254 Texas 10.0.0.25 80 HTTP UDP 2294 +2010-12-01 23:18:03 Atlanta 10.0.0.1 59491 Texas 10.0.0.29 80 HTTP UDP 2963 +2010-12-01 23:18:04 Atlanta 10.0.0.1 61682 Texas 10.0.0.26 80 HTTP UDP 3589 +2010-12-01 23:18:05 Atlanta 10.0.0.1 52632 Texas 10.0.0.27 80 HTTP UDP 5899 +2010-12-01 23:18:06 Atlanta 10.0.0.1 62788 Texas 10.0.0.26 80 HTTP UDP 5556 +2010-12-01 23:18:07 Atlanta 10.0.0.1 57966 Texas 10.0.0.27 80 HTTP UDP 4150 +2010-12-01 23:18:08 Atlanta 10.0.0.1 56259 Texas 10.0.0.25 80 HTTP UDP 4407 +2010-12-01 23:18:09 Atlanta 10.0.0.1 56828 Texas 10.0.0.26 80 HTTP UDP 2867 +2010-12-01 23:18:10 Atlanta 10.0.0.1 54266 Texas 10.0.0.27 80 HTTP UDP 5217 +2010-12-01 23:18:11 Atlanta 10.0.0.1 54504 Texas 10.0.0.25 80 HTTP UDP 5804 +2010-12-01 23:18:12 Atlanta 10.0.0.1 52367 Texas 10.0.0.28 80 HTTP UDP 3959 +2010-12-01 23:18:13 Atlanta 10.0.0.1 52980 Texas 10.0.0.28 80 HTTP UDP 2442 +2010-12-01 23:18:14 Atlanta 10.0.0.1 54330 Texas 10.0.0.26 80 HTTP UDP 2664 +2010-12-01 23:18:15 Atlanta 10.0.0.1 62114 Texas 10.0.0.29 80 HTTP UDP 2664 +2010-12-01 23:18:16 Atlanta 10.0.0.1 56448 Texas 10.0.0.28 80 HTTP UDP 5989 +2010-12-01 23:18:17 Atlanta 10.0.0.1 52797 Texas 10.0.0.29 80 HTTP UDP 2620 +2010-12-01 23:18:18 Atlanta 10.0.0.1 58372 Texas 10.0.0.29 80 HTTP UDP 4093 +2010-12-01 23:18:19 Atlanta 10.0.0.1 56107 Texas 10.0.0.26 80 HTTP UDP 4802 +2010-12-01 23:18:20 Atlanta 10.0.0.1 50688 Texas 10.0.0.29 80 HTTP UDP 4374 +2010-12-01 23:18:21 Atlanta 10.0.0.1 51719 Texas 10.0.0.27 80 HTTP UDP 5917 +2010-12-01 23:18:22 Atlanta 10.0.0.1 64783 Texas 10.0.0.30 80 HTTP UDP 4416 +2010-12-01 23:18:23 Atlanta 10.0.0.1 54900 Texas 10.0.0.29 80 HTTP UDP 2684 +2010-12-01 23:18:24 Atlanta 10.0.0.1 62594 Texas 10.0.0.28 80 HTTP UDP 3144 +2010-12-01 23:18:25 Atlanta 10.0.0.1 56100 Texas 10.0.0.27 80 HTTP UDP 5169 +2010-12-01 23:18:26 Atlanta 10.0.0.1 63178 Texas 10.0.0.28 80 HTTP UDP 2757 +2010-12-01 23:18:27 Atlanta 10.0.0.1 54476 Texas 10.0.0.25 80 HTTP UDP 4886 +2010-12-01 23:18:28 Atlanta 10.0.0.1 62977 Texas 10.0.0.30 80 HTTP UDP 5133 +2010-12-01 23:18:29 Atlanta 10.0.0.1 63185 Texas 10.0.0.25 80 HTTP UDP 4038 +2010-12-01 23:18:30 Atlanta 10.0.0.1 55279 Texas 10.0.0.27 80 HTTP UDP 3211 +2010-12-01 23:18:31 Atlanta 10.0.0.1 53165 Texas 10.0.0.29 80 HTTP UDP 2620 +2010-12-01 23:18:32 Atlanta 10.0.0.1 61307 Texas 10.0.0.25 80 HTTP UDP 5019 +2010-12-01 23:18:33 Atlanta 10.0.0.1 55429 Texas 10.0.0.26 80 HTTP UDP 2988 +2010-12-01 23:18:34 Atlanta 10.0.0.1 56468 Texas 10.0.0.29 80 HTTP UDP 5963 +2010-12-01 23:18:35 Atlanta 10.0.0.1 62774 Texas 10.0.0.27 80 HTTP UDP 2088 +2010-12-01 23:18:36 Atlanta 10.0.0.1 61912 Texas 10.0.0.25 80 HTTP UDP 2432 +2010-12-01 23:18:37 Atlanta 10.0.0.1 62295 Texas 10.0.0.27 80 HTTP UDP 3733 +2010-12-01 23:18:38 Atlanta 10.0.0.1 65133 Texas 10.0.0.25 80 HTTP UDP 5930 +2010-12-01 23:18:39 Atlanta 10.0.0.1 60954 Texas 10.0.0.27 80 HTTP UDP 5584 +2010-12-01 23:18:40 Atlanta 10.0.0.1 49924 Texas 10.0.0.27 80 HTTP UDP 5787 +2010-12-01 23:18:41 Atlanta 10.0.0.1 51380 Texas 10.0.0.25 80 HTTP UDP 4416 +2010-12-01 23:18:42 Atlanta 10.0.0.1 52825 Texas 10.0.0.26 80 HTTP UDP 5231 +2010-12-01 23:18:43 Atlanta 10.0.0.1 50938 Texas 10.0.0.27 80 HTTP UDP 4359 +2010-12-01 23:18:44 Atlanta 10.0.0.1 52093 Texas 10.0.0.28 80 HTTP UDP 4008 +2010-12-01 23:18:45 Atlanta 10.0.0.1 50393 Texas 10.0.0.25 80 HTTP UDP 2265 +2010-12-01 23:18:46 Atlanta 10.0.0.1 53094 Texas 10.0.0.25 80 HTTP UDP 5668 +2010-12-01 23:18:47 Atlanta 10.0.0.1 56090 Texas 10.0.0.28 80 HTTP UDP 2255 +2010-12-01 23:18:48 Atlanta 10.0.0.1 58249 Texas 10.0.0.25 80 HTTP UDP 4774 +2010-12-01 23:18:49 Atlanta 10.0.0.1 59558 Texas 10.0.0.29 80 HTTP UDP 3875 +2010-12-01 23:18:50 Atlanta 10.0.0.1 52666 Texas 10.0.0.27 80 HTTP UDP 5340 +2010-12-01 23:18:51 Atlanta 10.0.0.1 53799 Texas 10.0.0.27 80 HTTP UDP 4121 +2010-12-01 23:18:52 Atlanta 10.0.0.1 54073 Texas 10.0.0.28 80 HTTP UDP 5952 +2010-12-01 23:18:53 Atlanta 10.0.0.1 56827 Texas 10.0.0.29 80 HTTP UDP 4399 +2010-12-01 23:18:54 Atlanta 10.0.0.1 54345 Texas 10.0.0.25 80 HTTP UDP 3788 +2010-12-01 23:18:55 Atlanta 10.0.0.1 51464 Texas 10.0.0.30 80 HTTP UDP 3317 +2010-12-01 23:18:56 Atlanta 10.0.0.1 56211 Texas 10.0.0.27 80 HTTP UDP 3814 +2010-12-01 23:18:57 Atlanta 10.0.0.1 60573 Texas 10.0.0.29 80 HTTP UDP 5436 +2010-12-01 23:18:58 Atlanta 10.0.0.1 59162 Texas 10.0.0.27 80 HTTP UDP 2944 +2010-12-01 23:18:59 Atlanta 10.0.0.1 59131 Texas 10.0.0.28 80 HTTP UDP 3628 +2010-12-01 23:19:00 Atlanta 10.0.0.1 62210 Texas 10.0.0.28 80 HTTP UDP 2242 +2010-12-01 23:19:01 Atlanta 10.0.0.1 58460 Texas 10.0.0.28 80 HTTP UDP 4238 +2010-12-01 23:19:02 Atlanta 10.0.0.1 55407 Texas 10.0.0.27 80 HTTP UDP 3136 +2010-12-01 23:19:03 Atlanta 10.0.0.1 64729 Texas 10.0.0.30 80 HTTP UDP 2263 +2010-12-01 23:19:04 Atlanta 10.0.0.1 61438 Texas 10.0.0.25 80 HTTP UDP 2226 +2010-12-01 23:19:05 Atlanta 10.0.0.1 56830 Texas 10.0.0.26 80 HTTP UDP 5817 +2010-12-01 23:19:06 Atlanta 10.0.0.1 63611 Texas 10.0.0.28 80 HTTP UDP 2771 +2010-12-01 23:19:07 Atlanta 10.0.0.1 49267 Texas 10.0.0.29 80 HTTP UDP 5500 +2010-12-01 23:19:08 Atlanta 10.0.0.1 65485 Texas 10.0.0.30 80 HTTP UDP 4414 +2010-12-01 23:19:09 Atlanta 10.0.0.1 62220 Texas 10.0.0.26 80 HTTP UDP 4160 +2010-12-01 23:19:10 Atlanta 10.0.0.1 58296 Texas 10.0.0.26 80 HTTP UDP 4077 +2010-12-01 23:19:11 Atlanta 10.0.0.1 58258 Texas 10.0.0.28 80 HTTP UDP 5558 +2010-12-01 23:19:12 Atlanta 10.0.0.1 49484 Texas 10.0.0.30 80 HTTP UDP 2318 +2010-12-01 23:19:13 Atlanta 10.0.0.1 65321 Texas 10.0.0.26 80 HTTP UDP 5907 +2010-12-01 23:19:14 Atlanta 10.0.0.1 53955 Texas 10.0.0.26 80 HTTP UDP 5511 +2010-12-01 23:19:15 Atlanta 10.0.0.1 62119 Texas 10.0.0.26 80 HTTP UDP 4103 +2010-12-01 23:19:16 Atlanta 10.0.0.1 54155 Texas 10.0.0.27 80 HTTP UDP 3325 +2010-12-01 23:19:17 Atlanta 10.0.0.1 52860 Texas 10.0.0.25 80 HTTP UDP 4705 +2010-12-01 23:19:18 Atlanta 10.0.0.1 61504 Texas 10.0.0.28 80 HTTP UDP 4636 +2010-12-01 23:19:19 Atlanta 10.0.0.1 60803 Texas 10.0.0.29 80 HTTP UDP 2318 +2010-12-01 23:19:20 Atlanta 10.0.0.1 60113 Texas 10.0.0.27 80 HTTP UDP 4955 +2010-12-01 23:19:21 Atlanta 10.0.0.1 61110 Texas 10.0.0.30 80 HTTP UDP 4043 +2010-12-01 23:19:22 Atlanta 10.0.0.1 62745 Texas 10.0.0.28 80 HTTP UDP 2684 +2010-12-01 23:19:23 Atlanta 10.0.0.1 58771 Texas 10.0.0.25 80 HTTP UDP 4289 +2010-12-01 23:19:24 Atlanta 10.0.0.1 59101 Texas 10.0.0.27 80 HTTP UDP 3419 +2010-12-01 23:19:25 Atlanta 10.0.0.1 56916 Texas 10.0.0.27 80 HTTP UDP 4025 +2010-12-01 23:19:26 Atlanta 10.0.0.1 64899 Texas 10.0.0.26 80 HTTP UDP 4090 +2010-12-01 23:19:27 Atlanta 10.0.0.1 53508 Texas 10.0.0.29 80 HTTP UDP 5014 +2010-12-01 23:19:28 Atlanta 10.0.0.1 54879 Texas 10.0.0.25 80 HTTP UDP 5759 +2010-12-01 23:19:29 Atlanta 10.0.0.1 52033 Texas 10.0.0.27 80 HTTP UDP 3686 +2010-12-01 23:19:30 Atlanta 10.0.0.1 60583 Texas 10.0.0.28 80 HTTP UDP 2095 +2010-12-01 23:19:31 Atlanta 10.0.0.1 55929 Texas 10.0.0.29 80 HTTP UDP 2082 +2010-12-01 23:19:32 Atlanta 10.0.0.1 55070 Texas 10.0.0.28 80 HTTP UDP 3174 +2010-12-01 23:19:33 Atlanta 10.0.0.1 56477 Texas 10.0.0.25 80 HTTP UDP 4502 +2010-12-01 23:19:34 Atlanta 10.0.0.1 57740 Texas 10.0.0.26 80 HTTP UDP 2579 +2010-12-01 23:19:35 Atlanta 10.0.0.1 50350 Texas 10.0.0.27 80 HTTP UDP 5444 +2010-12-01 23:19:36 Atlanta 10.0.0.1 61957 Texas 10.0.0.30 80 HTTP UDP 4164 +2010-12-01 23:19:37 Atlanta 10.0.0.1 53547 Texas 10.0.0.28 80 HTTP UDP 3736 +2010-12-01 23:19:38 Atlanta 10.0.0.1 61706 Texas 10.0.0.26 80 HTTP UDP 2934 +2010-12-01 23:19:39 Atlanta 10.0.0.1 64751 Texas 10.0.0.25 80 HTTP UDP 5472 +2010-12-01 23:19:40 Atlanta 10.0.0.1 57869 Texas 10.0.0.25 80 HTTP UDP 3205 +2010-12-01 23:19:41 Atlanta 10.0.0.1 57108 Texas 10.0.0.28 80 HTTP UDP 3970 +2010-12-01 23:19:42 Atlanta 10.0.0.1 54203 Texas 10.0.0.25 80 HTTP UDP 4189 +2010-12-01 23:19:43 Atlanta 10.0.0.1 61627 Texas 10.0.0.28 80 HTTP UDP 4496 +2010-12-01 23:19:44 Atlanta 10.0.0.1 54564 Texas 10.0.0.26 80 HTTP UDP 2642 +2010-12-01 23:19:45 Atlanta 10.0.0.1 55407 Texas 10.0.0.25 80 HTTP UDP 4598 +2010-12-01 23:19:46 Atlanta 10.0.0.1 62918 Texas 10.0.0.27 80 HTTP UDP 5500 +2010-12-01 23:19:47 Atlanta 10.0.0.1 64661 Texas 10.0.0.26 80 HTTP UDP 2881 +2010-12-01 23:19:48 Atlanta 10.0.0.1 56786 Texas 10.0.0.29 80 HTTP UDP 2511 +2010-12-01 23:19:49 Atlanta 10.0.0.1 60816 Texas 10.0.0.28 80 HTTP UDP 5948 +2010-12-01 23:19:50 Atlanta 10.0.0.1 58387 Texas 10.0.0.28 80 HTTP UDP 3365 +2010-12-01 23:19:51 Atlanta 10.0.0.1 50384 Texas 10.0.0.30 80 HTTP UDP 3486 +2010-12-01 23:19:52 Atlanta 10.0.0.1 64719 Texas 10.0.0.25 80 HTTP UDP 3114 +2010-12-01 23:19:53 Atlanta 10.0.0.1 50362 Texas 10.0.0.28 80 HTTP UDP 3466 +2010-12-01 23:19:54 Atlanta 10.0.0.1 60372 Texas 10.0.0.28 80 HTTP UDP 2963 +2010-12-01 23:19:55 Atlanta 10.0.0.1 57346 Texas 10.0.0.26 80 HTTP UDP 3398 +2010-12-01 23:19:56 Atlanta 10.0.0.1 65078 Texas 10.0.0.25 80 HTTP UDP 2790 +2010-12-01 23:19:57 Atlanta 10.0.0.1 55222 Texas 10.0.0.28 80 HTTP UDP 3945 +2010-12-01 23:19:58 Atlanta 10.0.0.1 65066 Texas 10.0.0.29 80 HTTP UDP 5907 +2010-12-01 23:19:59 Atlanta 10.0.0.1 56670 Texas 10.0.0.27 80 HTTP UDP 5583 +2010-12-01 23:20:00 Atlanta 10.0.0.1 58130 Texas 10.0.0.26 80 HTTP UDP 3522 +2010-12-01 23:20:01 Atlanta 10.0.0.1 56001 Texas 10.0.0.25 80 HTTP UDP 5952 +2010-12-01 23:20:02 Atlanta 10.0.0.1 60451 Texas 10.0.0.30 80 HTTP UDP 2656 +2010-12-01 23:20:03 Atlanta 10.0.0.1 64523 Texas 10.0.0.25 80 HTTP UDP 5057 +2010-12-01 23:20:04 Atlanta 10.0.0.1 62059 Texas 10.0.0.28 80 HTTP UDP 5472 +2010-12-01 23:20:05 Atlanta 10.0.0.1 53634 Texas 10.0.0.27 80 HTTP UDP 5761 +2010-12-01 23:20:06 Atlanta 10.0.0.1 49773 Texas 10.0.0.27 80 HTTP UDP 4867 +2010-12-01 23:20:07 Atlanta 10.0.0.1 52401 Texas 10.0.0.28 80 HTTP UDP 3009 +2010-12-01 23:20:08 Atlanta 10.0.0.1 61966 Texas 10.0.0.25 80 HTTP UDP 5997 +2010-12-01 23:20:09 Atlanta 10.0.0.1 62378 Texas 10.0.0.29 80 HTTP UDP 5705 +2010-12-01 23:20:10 Atlanta 10.0.0.1 53441 Texas 10.0.0.28 80 HTTP UDP 4171 +2010-12-01 23:20:11 Atlanta 10.0.0.1 55248 Texas 10.0.0.29 80 HTTP UDP 3147 +2010-12-01 23:20:12 Atlanta 10.0.0.1 56035 Texas 10.0.0.29 80 HTTP UDP 3581 +2010-12-01 23:20:13 Atlanta 10.0.0.1 51853 Texas 10.0.0.25 80 HTTP UDP 2067 +2010-12-01 23:20:14 Atlanta 10.0.0.1 55246 Texas 10.0.0.27 80 HTTP UDP 3142 +2010-12-01 23:20:15 Atlanta 10.0.0.1 52883 Texas 10.0.0.25 80 HTTP UDP 2910 +2010-12-01 23:20:16 Atlanta 10.0.0.1 65448 Texas 10.0.0.30 80 HTTP UDP 3855 +2010-12-01 23:20:17 Atlanta 10.0.0.1 50840 Texas 10.0.0.27 80 HTTP UDP 3581 +2010-12-01 23:20:18 Atlanta 10.0.0.1 62877 Texas 10.0.0.28 80 HTTP UDP 2857 +2010-12-01 23:20:19 Atlanta 10.0.0.1 49519 Texas 10.0.0.25 80 HTTP UDP 2215 +2010-12-01 23:20:20 Atlanta 10.0.0.1 65376 Texas 10.0.0.30 80 HTTP UDP 3331 +2010-12-01 23:20:21 Atlanta 10.0.0.1 53087 Texas 10.0.0.27 80 HTTP UDP 3581 +2010-12-01 23:20:22 Atlanta 10.0.0.1 61915 Texas 10.0.0.28 80 HTTP UDP 3851 +2010-12-01 23:20:23 Atlanta 10.0.0.1 54563 Texas 10.0.0.25 80 HTTP UDP 3261 +2010-12-01 23:20:24 Atlanta 10.0.0.1 64416 Texas 10.0.0.28 80 HTTP UDP 4135 +2010-12-01 23:20:25 Atlanta 10.0.0.1 50332 Texas 10.0.0.26 80 HTTP UDP 2007 +2010-12-01 23:20:26 Atlanta 10.0.0.1 52844 Texas 10.0.0.26 80 HTTP UDP 3017 +2010-12-01 23:20:27 Atlanta 10.0.0.1 53096 Texas 10.0.0.26 80 HTTP UDP 4324 +2010-12-01 23:20:28 Atlanta 10.0.0.1 62292 Texas 10.0.0.25 80 HTTP UDP 3509 +2010-12-01 23:20:29 Atlanta 10.0.0.1 51375 Texas 10.0.0.26 80 HTTP UDP 5501 +2010-12-01 23:20:30 Atlanta 10.0.0.1 62456 Texas 10.0.0.25 80 HTTP UDP 2189 +2010-12-01 23:20:31 Atlanta 10.0.0.1 56969 Texas 10.0.0.25 80 HTTP UDP 2656 +2010-12-01 23:20:32 Atlanta 10.0.0.1 61623 Texas 10.0.0.28 80 HTTP UDP 4692 +2010-12-01 23:20:33 Atlanta 10.0.0.1 56884 Texas 10.0.0.27 80 HTTP UDP 5632 +2010-12-01 23:20:34 Atlanta 10.0.0.1 54301 Texas 10.0.0.25 80 HTTP UDP 5593 +2010-12-01 23:20:35 Atlanta 10.0.0.1 58557 Texas 10.0.0.29 80 HTTP UDP 3509 +2010-12-01 23:20:36 Atlanta 10.0.0.1 56965 Texas 10.0.0.26 80 HTTP UDP 5533 +2010-12-01 23:20:37 Atlanta 10.0.0.1 65167 Texas 10.0.0.25 80 HTTP UDP 5380 +2010-12-01 23:20:38 Atlanta 10.0.0.1 61587 Texas 10.0.0.26 80 HTTP UDP 3872 +2010-12-01 23:20:39 Atlanta 10.0.0.1 53548 Texas 10.0.0.27 80 HTTP UDP 5284 +2010-12-01 23:20:40 Atlanta 10.0.0.1 51092 Texas 10.0.0.30 80 HTTP UDP 4369 +2010-12-01 23:20:41 Atlanta 10.0.0.1 52450 Texas 10.0.0.27 80 HTTP UDP 3969 +2010-12-01 23:20:42 Atlanta 10.0.0.1 65439 Texas 10.0.0.25 80 HTTP UDP 2002 +2010-12-01 23:20:43 Atlanta 10.0.0.1 62216 Texas 10.0.0.27 80 HTTP UDP 5804 +2010-12-01 23:20:44 Atlanta 10.0.0.1 50310 Texas 10.0.0.29 80 HTTP UDP 3486 +2010-12-01 23:20:45 Atlanta 10.0.0.1 50165 Texas 10.0.0.30 80 HTTP UDP 2003 +2010-12-01 23:20:46 Atlanta 10.0.0.1 55213 Texas 10.0.0.28 80 HTTP UDP 5119 +2010-12-01 23:20:47 Atlanta 10.0.0.1 51632 Texas 10.0.0.27 80 HTTP UDP 5098 +2010-12-01 23:20:48 Atlanta 10.0.0.1 53839 Texas 10.0.0.29 80 HTTP UDP 2694 +2010-12-01 23:20:49 Atlanta 10.0.0.1 60448 Texas 10.0.0.25 80 HTTP UDP 5558 +2010-12-01 23:20:50 Atlanta 10.0.0.1 56251 Texas 10.0.0.27 80 HTTP UDP 3118 +2010-12-01 23:20:51 Atlanta 10.0.0.1 62063 Texas 10.0.0.27 80 HTTP UDP 4193 +2010-12-01 23:20:52 Atlanta 10.0.0.1 64434 Texas 10.0.0.25 80 HTTP UDP 2850 +2010-12-01 23:20:53 Atlanta 10.0.0.1 60316 Texas 10.0.0.27 80 HTTP UDP 3052 +2010-12-01 23:20:54 Atlanta 10.0.0.1 52371 Texas 10.0.0.29 80 HTTP UDP 3871 +2010-12-01 23:20:55 Atlanta 10.0.0.1 60332 Texas 10.0.0.25 80 HTTP UDP 2308 +2010-12-01 23:20:56 Atlanta 10.0.0.1 53026 Texas 10.0.0.28 80 HTTP UDP 4546 +2010-12-01 23:20:57 Atlanta 10.0.0.1 62071 Texas 10.0.0.29 80 HTTP UDP 3571 +2010-12-01 23:20:58 Atlanta 10.0.0.1 51403 Texas 10.0.0.25 80 HTTP UDP 4289 +2010-12-01 23:20:59 Atlanta 10.0.0.1 63667 Texas 10.0.0.25 80 HTTP UDP 3908 +2010-12-01 23:21:00 Atlanta 10.0.0.1 63758 Texas 10.0.0.30 80 HTTP UDP 3142 +2010-12-01 23:21:01 Atlanta 10.0.0.1 63537 Texas 10.0.0.26 80 HTTP UDP 2088 +2010-12-01 23:21:02 Atlanta 10.0.0.1 50728 Texas 10.0.0.25 80 HTTP UDP 2361 +2010-12-01 23:21:03 Atlanta 10.0.0.1 51782 Texas 10.0.0.29 80 HTTP UDP 4449 +2010-12-01 23:21:04 Atlanta 10.0.0.1 51519 Texas 10.0.0.28 80 HTTP UDP 5163 +2010-12-01 23:21:05 Atlanta 10.0.0.1 62273 Texas 10.0.0.29 80 HTTP UDP 2988 +2010-12-01 23:21:06 Atlanta 10.0.0.1 65416 Texas 10.0.0.26 80 HTTP UDP 5917 +2010-12-01 23:21:07 Atlanta 10.0.0.1 49505 Texas 10.0.0.25 80 HTTP UDP 4414 +2010-12-01 23:21:08 Atlanta 10.0.0.1 53516 Texas 10.0.0.28 80 HTTP UDP 3066 +2010-12-01 23:21:09 Atlanta 10.0.0.1 57990 Texas 10.0.0.29 80 HTTP UDP 4959 +2010-12-01 23:21:10 Atlanta 10.0.0.1 59603 Texas 10.0.0.25 80 HTTP UDP 5098 +2010-12-01 23:21:11 Atlanta 10.0.0.1 56556 Texas 10.0.0.29 80 HTTP UDP 2850 +2010-12-01 23:21:12 Atlanta 10.0.0.1 53305 Texas 10.0.0.25 80 HTTP UDP 2255 +2010-12-01 23:21:13 Atlanta 10.0.0.1 52956 Texas 10.0.0.28 80 HTTP UDP 3017 +2010-12-01 23:21:14 Atlanta 10.0.0.1 54706 Texas 10.0.0.28 80 HTTP UDP 2579 +2010-12-01 23:21:15 Atlanta 10.0.0.1 52425 Texas 10.0.0.28 80 HTTP UDP 2694 +2010-12-01 23:21:16 Atlanta 10.0.0.1 65320 Texas 10.0.0.25 80 HTTP UDP 4251 +2010-12-01 23:21:17 Atlanta 10.0.0.1 56129 Texas 10.0.0.26 80 HTTP UDP 5500 +2010-12-01 23:21:18 Atlanta 10.0.0.1 53320 Texas 10.0.0.25 80 HTTP UDP 3572 +2010-12-01 23:21:19 Atlanta 10.0.0.1 60240 Texas 10.0.0.28 80 HTTP UDP 3589 +2010-12-01 23:21:20 Atlanta 10.0.0.1 57555 Texas 10.0.0.28 80 HTTP UDP 4827 +2010-12-01 23:21:21 Atlanta 10.0.0.1 55374 Texas 10.0.0.28 80 HTTP UDP 4393 +2010-12-01 23:21:22 Atlanta 10.0.0.1 51721 Texas 10.0.0.30 80 HTTP UDP 4395 +2010-12-01 23:21:23 Atlanta 10.0.0.1 61518 Texas 10.0.0.29 80 HTTP UDP 4646 +2010-12-01 23:21:24 Atlanta 10.0.0.1 64476 Texas 10.0.0.25 80 HTTP UDP 5938 +2010-12-01 23:21:25 Atlanta 10.0.0.1 62271 Texas 10.0.0.25 80 HTTP UDP 2263 +2010-12-01 23:21:26 Atlanta 10.0.0.1 60174 Texas 10.0.0.25 80 HTTP UDP 5369 +2010-12-01 23:21:27 Atlanta 10.0.0.1 56960 Texas 10.0.0.28 80 HTTP UDP 4414 +2010-12-01 23:21:28 Atlanta 10.0.0.1 53365 Texas 10.0.0.27 80 HTTP UDP 2830 +2010-12-01 23:21:29 Atlanta 10.0.0.1 59754 Texas 10.0.0.29 80 HTTP UDP 5133 +2010-12-01 23:21:30 Atlanta 10.0.0.1 59208 Texas 10.0.0.27 80 HTTP UDP 3970 +2010-12-01 23:21:31 Atlanta 10.0.0.1 55260 Texas 10.0.0.25 80 HTTP UDP 4918 +2010-12-01 23:21:32 Atlanta 10.0.0.1 51537 Texas 10.0.0.26 80 HTTP UDP 5422 +2010-12-01 23:21:33 Atlanta 10.0.0.1 58348 Texas 10.0.0.28 80 HTTP UDP 2693 +2010-12-01 23:21:34 Atlanta 10.0.0.1 55064 Texas 10.0.0.29 80 HTTP UDP 3074 +2010-12-01 23:21:35 Atlanta 10.0.0.1 60796 Texas 10.0.0.28 80 HTTP UDP 5232 +2010-12-01 23:21:36 Atlanta 10.0.0.1 53781 Texas 10.0.0.25 80 HTTP UDP 2804 +2010-12-01 23:21:37 Atlanta 10.0.0.1 60934 Texas 10.0.0.27 80 HTTP UDP 2378 +2010-12-01 23:21:38 Atlanta 10.0.0.1 55563 Texas 10.0.0.30 80 HTTP UDP 4802 +2010-12-01 23:21:39 Atlanta 10.0.0.1 64517 Texas 10.0.0.30 80 HTTP UDP 3559 +2010-12-01 23:21:40 Atlanta 10.0.0.1 55232 Texas 10.0.0.30 80 HTTP UDP 3940 +2010-12-01 23:21:41 Atlanta 10.0.0.1 50218 Texas 10.0.0.25 80 HTTP UDP 5548 +2010-12-01 23:21:42 Atlanta 10.0.0.1 61794 Texas 10.0.0.30 80 HTTP UDP 2447 +2010-12-01 23:21:43 Atlanta 10.0.0.1 60615 Texas 10.0.0.30 80 HTTP UDP 5135 +2010-12-01 23:21:44 Atlanta 10.0.0.1 56896 Texas 10.0.0.29 80 HTTP UDP 4903 +2010-12-01 23:21:45 Atlanta 10.0.0.1 58688 Texas 10.0.0.28 80 HTTP UDP 2576 +2010-12-01 23:21:46 Atlanta 10.0.0.1 62204 Texas 10.0.0.28 80 HTTP UDP 2355 +2010-12-01 23:21:47 Atlanta 10.0.0.1 54062 Texas 10.0.0.29 80 HTTP UDP 5917 +2010-12-01 23:21:48 Atlanta 10.0.0.1 56662 Texas 10.0.0.27 80 HTTP UDP 4955 +2010-12-01 23:21:49 Atlanta 10.0.0.1 49680 Texas 10.0.0.26 80 HTTP UDP 3509 +2010-12-01 23:21:50 Atlanta 10.0.0.1 63414 Texas 10.0.0.29 80 HTTP UDP 3540 +2010-12-01 23:21:51 Atlanta 10.0.0.1 50462 Texas 10.0.0.27 80 HTTP UDP 4646 +2010-12-01 23:21:52 Atlanta 10.0.0.1 55132 Texas 10.0.0.27 80 HTTP UDP 5460 +2010-12-01 23:21:53 Atlanta 10.0.0.1 55003 Texas 10.0.0.30 80 HTTP UDP 5255 +2010-12-01 23:21:54 Atlanta 10.0.0.1 60665 Texas 10.0.0.25 80 HTTP UDP 2921 +2010-12-01 23:21:55 Atlanta 10.0.0.1 51028 Texas 10.0.0.26 80 HTTP UDP 3147 +2010-12-01 23:21:56 Atlanta 10.0.0.1 51792 Texas 10.0.0.26 80 HTTP UDP 3032 +2010-12-01 23:21:57 Atlanta 10.0.0.1 60711 Texas 10.0.0.27 80 HTTP UDP 4774 +2010-12-01 23:21:58 Atlanta 10.0.0.1 62874 Texas 10.0.0.28 80 HTTP UDP 3016 +2010-12-01 23:21:59 Atlanta 10.0.0.1 49520 Texas 10.0.0.29 80 HTTP UDP 4102 +2010-12-01 23:22:00 Atlanta 10.0.0.1 63399 Texas 10.0.0.25 80 HTTP UDP 3182 +2010-12-01 23:22:01 Atlanta 10.0.0.1 58231 Texas 10.0.0.29 80 HTTP UDP 5313 +2010-12-01 23:22:02 Atlanta 10.0.0.1 51327 Texas 10.0.0.30 80 HTTP UDP 4627 +2010-12-01 23:22:03 Atlanta 10.0.0.1 63306 Texas 10.0.0.29 80 HTTP UDP 2553 +2010-12-01 23:22:04 Atlanta 10.0.0.1 65410 Texas 10.0.0.28 80 HTTP UDP 2308 +2010-12-01 23:22:05 Atlanta 10.0.0.1 55211 Texas 10.0.0.30 80 HTTP UDP 3138 +2010-12-01 23:22:06 Atlanta 10.0.0.1 50343 Texas 10.0.0.27 80 HTTP UDP 3995 +2010-12-01 23:22:07 Atlanta 10.0.0.1 62606 Texas 10.0.0.26 80 HTTP UDP 4192 +2010-12-01 23:22:08 Atlanta 10.0.0.1 50868 Texas 10.0.0.26 80 HTTP UDP 2462 +2010-12-01 23:22:09 Atlanta 10.0.0.1 65211 Texas 10.0.0.27 80 HTTP UDP 5527 +2010-12-01 23:22:10 Atlanta 10.0.0.1 55691 Texas 10.0.0.29 80 HTTP UDP 5501 +2010-12-01 23:22:11 Atlanta 10.0.0.1 59062 Texas 10.0.0.30 80 HTTP UDP 3538 +2010-12-01 23:22:12 Atlanta 10.0.0.1 52156 Texas 10.0.0.27 80 HTTP UDP 3549 +2010-12-01 23:22:13 Atlanta 10.0.0.1 56912 Texas 10.0.0.25 80 HTTP UDP 3450 +2010-12-01 23:22:14 Atlanta 10.0.0.1 52754 Texas 10.0.0.26 80 HTTP UDP 3900 +2010-12-01 23:22:15 Atlanta 10.0.0.1 51476 Texas 10.0.0.29 80 HTTP UDP 2688 +2010-12-01 23:22:16 Atlanta 10.0.0.1 62438 Texas 10.0.0.28 80 HTTP UDP 4683 +2010-12-01 23:22:17 Atlanta 10.0.0.1 56958 Texas 10.0.0.28 80 HTTP UDP 5727 +2010-12-01 23:22:18 Atlanta 10.0.0.1 53490 Texas 10.0.0.26 80 HTTP UDP 2693 +2010-12-01 23:22:19 Atlanta 10.0.0.1 49277 Texas 10.0.0.26 80 HTTP UDP 5620 +2010-12-01 23:22:20 Atlanta 10.0.0.1 61146 Texas 10.0.0.29 80 HTTP UDP 3996 +2010-12-01 23:22:21 Atlanta 10.0.0.1 62708 Texas 10.0.0.28 80 HTTP UDP 5315 +2010-12-01 23:22:22 Atlanta 10.0.0.1 62062 Texas 10.0.0.25 80 HTTP UDP 4598 +2010-12-01 23:22:23 Atlanta 10.0.0.1 55280 Texas 10.0.0.26 80 HTTP UDP 3025 +2010-12-01 23:22:24 Atlanta 10.0.0.1 63264 Texas 10.0.0.25 80 HTTP UDP 4701 +2010-12-01 23:22:25 Atlanta 10.0.0.1 63540 Texas 10.0.0.27 80 HTTP UDP 4849 +2010-12-01 23:22:26 Atlanta 10.0.0.1 62474 Texas 10.0.0.27 80 HTTP UDP 2526 +2010-12-01 23:22:27 Atlanta 10.0.0.1 53990 Texas 10.0.0.30 80 HTTP UDP 2860 +2010-12-01 23:22:28 Atlanta 10.0.0.1 53021 Texas 10.0.0.29 80 HTTP UDP 3995 +2010-12-01 23:22:29 Atlanta 10.0.0.1 64423 Texas 10.0.0.26 80 HTTP UDP 5029 +2010-12-01 23:22:30 Atlanta 10.0.0.1 58439 Texas 10.0.0.27 80 HTTP UDP 2029 +2010-12-01 23:22:31 Atlanta 10.0.0.1 52222 Texas 10.0.0.30 80 HTTP UDP 5501 +2010-12-01 23:22:32 Atlanta 10.0.0.1 64264 Texas 10.0.0.25 80 HTTP UDP 2882 +2010-12-01 23:22:33 Atlanta 10.0.0.1 51298 Texas 10.0.0.30 80 HTTP UDP 5506 +2010-12-01 23:22:34 Atlanta 10.0.0.1 64058 Texas 10.0.0.28 80 HTTP UDP 4449 +2010-12-01 23:22:35 Atlanta 10.0.0.1 64724 Texas 10.0.0.26 80 HTTP UDP 5582 +2010-12-01 23:22:36 Atlanta 10.0.0.1 62015 Texas 10.0.0.25 80 HTTP UDP 2561 +2010-12-01 23:22:37 Atlanta 10.0.0.1 53689 Texas 10.0.0.25 80 HTTP UDP 5017 +2010-12-01 23:22:38 Atlanta 10.0.0.1 52023 Texas 10.0.0.25 80 HTTP UDP 5029 +2010-12-01 23:22:39 Atlanta 10.0.0.1 63856 Texas 10.0.0.29 80 HTTP UDP 5380 +2010-12-01 23:22:40 Atlanta 10.0.0.1 62511 Texas 10.0.0.28 80 HTTP UDP 2819 +2010-12-01 23:22:41 Atlanta 10.0.0.1 64801 Texas 10.0.0.30 80 HTTP UDP 5632 +2010-12-01 23:22:42 Atlanta 10.0.0.1 53288 Texas 10.0.0.28 80 HTTP UDP 5750 +2010-12-01 23:22:43 Atlanta 10.0.0.1 56871 Texas 10.0.0.27 80 HTTP UDP 3211 +2010-12-01 23:22:44 Atlanta 10.0.0.1 55108 Texas 10.0.0.25 80 HTTP UDP 5133 +2010-12-01 23:22:45 Atlanta 10.0.0.1 62248 Texas 10.0.0.28 80 HTTP UDP 3764 +2010-12-01 23:22:46 Atlanta 10.0.0.1 63088 Texas 10.0.0.28 80 HTTP UDP 2001 +2010-12-01 23:22:47 Atlanta 10.0.0.1 58267 Texas 10.0.0.27 80 HTTP UDP 3786 +2010-12-01 23:22:48 Atlanta 10.0.0.1 49776 Texas 10.0.0.25 80 HTTP UDP 5759 +2010-12-01 23:22:49 Atlanta 10.0.0.1 56545 Texas 10.0.0.27 80 HTTP UDP 3685 +2010-12-01 23:22:50 Atlanta 10.0.0.1 54094 Texas 10.0.0.28 80 HTTP UDP 5044 +2010-12-01 23:22:51 Atlanta 10.0.0.1 52028 Texas 10.0.0.25 80 HTTP UDP 3398 +2010-12-01 23:22:52 Atlanta 10.0.0.1 59306 Texas 10.0.0.26 80 HTTP UDP 5907 +2010-12-01 23:22:53 Atlanta 10.0.0.1 59633 Texas 10.0.0.26 80 HTTP UDP 4792 +2010-12-01 23:22:54 Atlanta 10.0.0.1 59443 Texas 10.0.0.28 80 HTTP UDP 2154 +2010-12-01 23:22:55 Atlanta 10.0.0.1 50125 Texas 10.0.0.30 80 HTTP UDP 4162 +2010-12-01 23:22:56 Atlanta 10.0.0.1 58640 Texas 10.0.0.28 80 HTTP UDP 5014 +2010-12-01 23:22:57 Atlanta 10.0.0.1 56022 Texas 10.0.0.26 80 HTTP UDP 4696 +2010-12-01 23:22:58 Atlanta 10.0.0.1 61888 Texas 10.0.0.29 80 HTTP UDP 5673 +2010-12-01 23:22:59 Atlanta 10.0.0.1 56989 Texas 10.0.0.29 80 HTTP UDP 5370 +2010-12-01 23:23:00 Atlanta 10.0.0.1 49308 Texas 10.0.0.29 80 HTTP UDP 2432 +2010-12-01 23:23:01 Atlanta 10.0.0.1 54167 Texas 10.0.0.29 80 HTTP UDP 2660 +2010-12-01 23:23:02 Atlanta 10.0.0.1 63482 Texas 10.0.0.30 80 HTTP UDP 5160 +2010-12-01 23:23:03 Atlanta 10.0.0.1 59422 Texas 10.0.0.27 80 HTTP UDP 3936 +2010-12-01 23:23:04 Atlanta 10.0.0.1 54387 Texas 10.0.0.25 80 HTTP UDP 5959 +2010-12-01 23:23:05 Atlanta 10.0.0.1 58114 Texas 10.0.0.29 80 HTTP UDP 3549 +2010-12-01 23:23:06 Atlanta 10.0.0.1 56290 Texas 10.0.0.27 80 HTTP UDP 3609 +2010-12-01 23:23:07 Atlanta 10.0.0.1 61584 Texas 10.0.0.25 80 HTTP UDP 3017 +2010-12-01 23:23:08 Atlanta 10.0.0.1 65334 Texas 10.0.0.27 80 HTTP UDP 2082 +2010-12-01 23:23:09 Atlanta 10.0.0.1 52895 Texas 10.0.0.28 80 HTTP UDP 4324 +2010-12-01 23:23:10 Atlanta 10.0.0.1 61499 Texas 10.0.0.25 80 HTTP UDP 3293 +2010-12-01 23:23:11 Atlanta 10.0.0.1 60170 Texas 10.0.0.30 80 HTTP UDP 3793 +2010-12-01 23:23:12 Atlanta 10.0.0.1 62298 Texas 10.0.0.29 80 HTTP UDP 5321 +2010-12-01 23:23:13 Atlanta 10.0.0.1 57617 Texas 10.0.0.26 80 HTTP UDP 3122 +2010-12-01 23:23:14 Atlanta 10.0.0.1 53701 Texas 10.0.0.28 80 HTTP UDP 4657 +2010-12-01 23:23:15 Atlanta 10.0.0.1 60620 Texas 10.0.0.26 80 HTTP UDP 3171 +2010-12-01 23:23:16 Atlanta 10.0.0.1 61181 Texas 10.0.0.25 80 HTTP UDP 3618 +2010-12-01 23:23:17 Atlanta 10.0.0.1 57081 Texas 10.0.0.26 80 HTTP UDP 5894 +2010-12-01 23:23:18 Atlanta 10.0.0.1 62227 Texas 10.0.0.30 80 HTTP UDP 4316 +2010-12-01 23:23:19 Atlanta 10.0.0.1 58477 Texas 10.0.0.26 80 HTTP UDP 5963 +2010-12-01 23:23:20 Atlanta 10.0.0.1 57010 Texas 10.0.0.26 80 HTTP UDP 5603 +2010-12-01 23:23:21 Atlanta 10.0.0.1 51907 Texas 10.0.0.30 80 HTTP UDP 5268 +2010-12-01 23:23:22 Atlanta 10.0.0.1 54899 Texas 10.0.0.25 80 HTTP UDP 3585 +2010-12-01 23:23:23 Atlanta 10.0.0.1 51946 Texas 10.0.0.28 80 HTTP UDP 2442 +2010-12-01 23:23:24 Atlanta 10.0.0.1 52342 Texas 10.0.0.30 80 HTTP UDP 5153 +2010-12-01 23:23:25 Atlanta 10.0.0.1 64876 Texas 10.0.0.27 80 HTTP UDP 5556 +2010-12-01 23:23:26 Atlanta 10.0.0.1 57341 Texas 10.0.0.27 80 HTTP UDP 5807 +2010-12-01 23:23:27 Atlanta 10.0.0.1 53139 Texas 10.0.0.27 80 HTTP UDP 5935 +2010-12-01 23:23:28 Atlanta 10.0.0.1 53717 Texas 10.0.0.27 80 HTTP UDP 3171 +2010-12-01 23:23:29 Atlanta 10.0.0.1 62481 Texas 10.0.0.26 80 HTTP UDP 4657 +2010-12-01 23:23:30 Atlanta 10.0.0.1 64095 Texas 10.0.0.27 80 HTTP UDP 4692 +2010-12-01 23:23:31 Atlanta 10.0.0.1 58431 Texas 10.0.0.25 80 HTTP UDP 2790 +2010-12-01 23:23:32 Atlanta 10.0.0.1 53817 Texas 10.0.0.28 80 HTTP UDP 3828 +2010-12-01 23:23:33 Atlanta 10.0.0.1 63054 Texas 10.0.0.30 80 HTTP UDP 3365 +2010-12-01 23:23:34 Atlanta 10.0.0.1 56412 Texas 10.0.0.28 80 HTTP UDP 4374 +2010-12-01 23:23:35 Atlanta 10.0.0.1 54657 Texas 10.0.0.29 80 HTTP UDP 5243 +2010-12-01 23:23:36 Atlanta 10.0.0.1 62621 Texas 10.0.0.26 80 HTTP UDP 5511 +2010-12-01 23:23:37 Atlanta 10.0.0.1 64045 Texas 10.0.0.30 80 HTTP UDP 3469 +2010-12-01 23:23:38 Atlanta 10.0.0.1 64127 Texas 10.0.0.27 80 HTTP UDP 4013 +2010-12-01 23:23:39 Atlanta 10.0.0.1 64173 Texas 10.0.0.27 80 HTTP UDP 2133 +2010-12-01 23:23:40 Atlanta 10.0.0.1 64756 Texas 10.0.0.30 80 HTTP UDP 2311 +2010-12-01 23:23:41 Atlanta 10.0.0.1 51824 Texas 10.0.0.30 80 HTTP UDP 5072 +2010-12-01 23:23:42 Atlanta 10.0.0.1 50588 Texas 10.0.0.29 80 HTTP UDP 3454 +2010-12-01 23:23:43 Atlanta 10.0.0.1 65018 Texas 10.0.0.29 80 HTTP UDP 4669 +2010-12-01 23:23:44 Atlanta 10.0.0.1 56390 Texas 10.0.0.28 80 HTTP UDP 2563 +2010-12-01 23:23:45 Atlanta 10.0.0.1 64899 Texas 10.0.0.25 80 HTTP UDP 5811 +2010-12-01 23:23:46 Atlanta 10.0.0.1 49517 Texas 10.0.0.26 80 HTTP UDP 2782 +2010-12-01 23:23:47 Atlanta 10.0.0.1 55421 Texas 10.0.0.27 80 HTTP UDP 3230 +2010-12-01 23:23:48 Atlanta 10.0.0.1 56981 Texas 10.0.0.27 80 HTTP UDP 3602 +2010-12-01 23:23:49 Atlanta 10.0.0.1 50030 Texas 10.0.0.26 80 HTTP UDP 2215 +2010-12-01 23:23:50 Atlanta 10.0.0.1 60334 Texas 10.0.0.29 80 HTTP UDP 4172 +2010-12-01 23:23:51 Atlanta 10.0.0.1 60409 Texas 10.0.0.30 80 HTTP UDP 3814 +2010-12-01 23:23:52 Atlanta 10.0.0.1 58159 Texas 10.0.0.30 80 HTTP UDP 5632 +2010-12-01 23:23:53 Atlanta 10.0.0.1 54450 Texas 10.0.0.29 80 HTTP UDP 3945 +2010-12-01 23:23:54 Atlanta 10.0.0.1 60434 Texas 10.0.0.28 80 HTTP UDP 3189 +2010-12-01 23:23:55 Atlanta 10.0.0.1 50551 Texas 10.0.0.25 80 HTTP UDP 5351 +2010-12-01 23:23:56 Atlanta 10.0.0.1 62654 Texas 10.0.0.27 80 HTTP UDP 5082 +2010-12-01 23:23:57 Atlanta 10.0.0.1 60476 Texas 10.0.0.26 80 HTTP UDP 5849 +2010-12-01 23:23:58 Atlanta 10.0.0.1 57857 Texas 10.0.0.30 80 HTTP UDP 3901 +2010-12-01 23:23:59 Atlanta 10.0.0.1 60182 Texas 10.0.0.27 80 HTTP UDP 2579 +2010-12-01 23:24:00 Atlanta 10.0.0.1 63768 Texas 10.0.0.26 80 HTTP UDP 2627 +2010-12-01 23:24:01 Atlanta 10.0.0.1 56533 Texas 10.0.0.25 80 HTTP UDP 2553 +2010-12-01 23:24:02 Atlanta 10.0.0.1 52404 Texas 10.0.0.29 80 HTTP UDP 2603 +2010-12-01 23:24:03 Atlanta 10.0.0.1 49921 Texas 10.0.0.30 80 HTTP UDP 4251 +2010-12-01 23:24:04 Atlanta 10.0.0.1 60692 Texas 10.0.0.25 80 HTTP UDP 5705 +2010-12-01 23:24:05 Atlanta 10.0.0.1 50393 Texas 10.0.0.28 80 HTTP UDP 3743 +2010-12-01 23:24:06 Atlanta 10.0.0.1 52037 Texas 10.0.0.28 80 HTTP UDP 2805 +2010-12-01 23:24:07 Atlanta 10.0.0.1 58774 Texas 10.0.0.28 80 HTTP UDP 3736 +2010-12-01 23:24:08 Atlanta 10.0.0.1 52754 Texas 10.0.0.29 80 HTTP UDP 4538 +2010-12-01 23:24:09 Atlanta 10.0.0.1 65274 Texas 10.0.0.25 80 HTTP UDP 5738 +2010-12-01 23:24:10 Atlanta 10.0.0.1 58148 Texas 10.0.0.27 80 HTTP UDP 3230 +2010-12-01 23:24:11 Atlanta 10.0.0.1 57179 Texas 10.0.0.27 80 HTTP UDP 4193 +2010-12-01 23:24:12 Atlanta 10.0.0.1 53910 Texas 10.0.0.28 80 HTTP UDP 4567 +2010-12-01 23:24:13 Atlanta 10.0.0.1 64203 Texas 10.0.0.27 80 HTTP UDP 5135 +2010-12-01 23:24:14 Atlanta 10.0.0.1 50580 Texas 10.0.0.26 80 HTTP UDP 2192 +2010-12-01 23:24:15 Atlanta 10.0.0.1 59761 Texas 10.0.0.29 80 HTTP UDP 3845 +2010-12-01 23:24:16 Atlanta 10.0.0.1 55943 Texas 10.0.0.30 80 HTTP UDP 4002 +2010-12-01 23:24:17 Atlanta 10.0.0.1 55075 Texas 10.0.0.29 80 HTTP UDP 4780 +2010-12-01 23:24:18 Atlanta 10.0.0.1 59641 Texas 10.0.0.30 80 HTTP UDP 5422 +2010-12-01 23:24:19 Atlanta 10.0.0.1 58418 Texas 10.0.0.25 80 HTTP UDP 3945 +2010-12-01 23:24:20 Atlanta 10.0.0.1 51949 Texas 10.0.0.27 80 HTTP UDP 3736 +2010-12-01 23:24:21 Atlanta 10.0.0.1 58685 Texas 10.0.0.27 80 HTTP UDP 3402 +2010-12-01 23:24:22 Atlanta 10.0.0.1 60483 Texas 10.0.0.30 80 HTTP UDP 5835 +2010-12-01 23:24:23 Atlanta 10.0.0.1 49389 Texas 10.0.0.30 80 HTTP UDP 3017 +2010-12-01 23:24:24 Atlanta 10.0.0.1 50913 Texas 10.0.0.28 80 HTTP UDP 2996 +2010-12-01 23:24:25 Atlanta 10.0.0.1 64928 Texas 10.0.0.26 80 HTTP UDP 2642 +2010-12-01 23:24:26 Atlanta 10.0.0.1 60659 Texas 10.0.0.28 80 HTTP UDP 2477 +2010-12-01 23:24:27 Atlanta 10.0.0.1 51696 Texas 10.0.0.25 80 HTTP UDP 2172 +2010-12-01 23:24:28 Atlanta 10.0.0.1 54032 Texas 10.0.0.27 80 HTTP UDP 4893 +2010-12-01 23:24:29 Atlanta 10.0.0.1 59329 Texas 10.0.0.28 80 HTTP UDP 4530 +2010-12-01 23:24:30 Atlanta 10.0.0.1 56745 Texas 10.0.0.28 80 HTTP UDP 3908 +2010-12-01 23:24:31 Atlanta 10.0.0.1 64759 Texas 10.0.0.25 80 HTTP UDP 4258 +2010-12-01 23:24:32 Atlanta 10.0.0.1 49212 Texas 10.0.0.28 80 HTTP UDP 4046 +2010-12-01 23:24:33 Atlanta 10.0.0.1 60199 Texas 10.0.0.29 80 HTTP UDP 4887 +2010-12-01 23:24:34 Atlanta 10.0.0.1 56994 Texas 10.0.0.29 80 HTTP UDP 3455 +2010-12-01 23:24:35 Atlanta 10.0.0.1 54419 Texas 10.0.0.28 80 HTTP UDP 5258 +2010-12-01 23:24:36 Atlanta 10.0.0.1 58120 Texas 10.0.0.28 80 HTTP UDP 2694 +2010-12-01 23:24:37 Atlanta 10.0.0.1 53390 Texas 10.0.0.29 80 HTTP UDP 3538 +2010-12-01 23:24:38 Atlanta 10.0.0.1 55264 Texas 10.0.0.29 80 HTTP UDP 3450 +2010-12-01 23:24:39 Atlanta 10.0.0.1 62290 Texas 10.0.0.27 80 HTTP UDP 5696 +2010-12-01 23:24:40 Atlanta 10.0.0.1 50683 Texas 10.0.0.26 80 HTTP UDP 4561 +2010-12-01 23:24:41 Atlanta 10.0.0.1 61735 Texas 10.0.0.30 80 HTTP UDP 3107 +2010-12-01 23:24:42 Atlanta 10.0.0.1 59785 Texas 10.0.0.30 80 HTTP UDP 2963 +2010-12-01 23:24:43 Atlanta 10.0.0.1 57303 Texas 10.0.0.27 80 HTTP UDP 5231 +2010-12-01 23:24:44 Atlanta 10.0.0.1 49744 Texas 10.0.0.30 80 HTTP UDP 4253 +2010-12-01 23:24:45 Atlanta 10.0.0.1 51957 Texas 10.0.0.25 80 HTTP UDP 5930 +2010-12-01 23:24:46 Atlanta 10.0.0.1 55646 Texas 10.0.0.30 80 HTTP UDP 3609 +2010-12-01 23:24:47 Atlanta 10.0.0.1 59251 Texas 10.0.0.26 80 HTTP UDP 3871 +2010-12-01 23:24:48 Atlanta 10.0.0.1 61988 Texas 10.0.0.27 80 HTTP UDP 4415 +2010-12-01 23:24:49 Atlanta 10.0.0.1 63865 Texas 10.0.0.29 80 HTTP UDP 3549 +2010-12-01 23:24:50 Atlanta 10.0.0.1 59934 Texas 10.0.0.27 80 HTTP UDP 5224 +2010-12-01 23:24:51 Atlanta 10.0.0.1 54264 Texas 10.0.0.26 80 HTTP UDP 2855 +2010-12-01 23:24:52 Atlanta 10.0.0.1 51408 Texas 10.0.0.30 80 HTTP UDP 3871 +2010-12-01 23:24:53 Atlanta 10.0.0.1 64458 Texas 10.0.0.30 80 HTTP UDP 5755 +2010-12-01 23:24:54 Atlanta 10.0.0.1 63915 Texas 10.0.0.29 80 HTTP UDP 2231 +2010-12-01 23:24:55 Atlanta 10.0.0.1 52704 Texas 10.0.0.26 80 HTTP UDP 3995 +2010-12-01 23:24:56 Atlanta 10.0.0.1 53316 Texas 10.0.0.26 80 HTTP UDP 3146 +2010-12-01 23:24:57 Atlanta 10.0.0.1 51151 Texas 10.0.0.30 80 HTTP UDP 4189 +2010-12-01 23:24:58 Atlanta 10.0.0.1 51015 Texas 10.0.0.26 80 HTTP UDP 4969 +2010-12-01 23:24:59 Atlanta 10.0.0.1 62723 Texas 10.0.0.27 80 HTTP UDP 5966 +2010-12-01 23:25:00 Atlanta 10.0.0.1 50970 Texas 10.0.0.25 80 HTTP UDP 4368 +2010-12-01 23:25:01 Atlanta 10.0.0.1 58938 Texas 10.0.0.30 80 HTTP UDP 5937 +2010-12-01 23:25:02 Atlanta 10.0.0.1 59036 Texas 10.0.0.25 80 HTTP UDP 3549 +2010-12-01 23:25:03 Atlanta 10.0.0.1 58209 Texas 10.0.0.27 80 HTTP UDP 2255 +2010-12-01 23:25:04 Atlanta 10.0.0.1 49695 Texas 10.0.0.27 80 HTTP UDP 2909 +2010-12-01 23:25:05 Atlanta 10.0.0.1 64318 Texas 10.0.0.25 80 HTTP UDP 5132 +2010-12-01 23:25:06 Atlanta 10.0.0.1 55529 Texas 10.0.0.27 80 HTTP UDP 4251 +2010-12-01 23:25:07 Atlanta 10.0.0.1 55021 Texas 10.0.0.27 80 HTTP UDP 2690 +2010-12-01 23:25:08 Atlanta 10.0.0.1 52794 Texas 10.0.0.29 80 HTTP UDP 4121 +2010-12-01 23:25:09 Atlanta 10.0.0.1 56927 Texas 10.0.0.25 80 HTTP UDP 4177 +2010-12-01 23:25:10 Atlanta 10.0.0.1 61542 Texas 10.0.0.28 80 HTTP UDP 3203 +2010-12-01 23:25:11 Atlanta 10.0.0.1 51914 Texas 10.0.0.26 80 HTTP UDP 3070 +2010-12-01 23:25:12 Atlanta 10.0.0.1 56539 Texas 10.0.0.30 80 HTTP UDP 5336 +2010-12-01 23:25:13 Atlanta 10.0.0.1 56632 Texas 10.0.0.25 80 HTTP UDP 5533 +2010-12-01 23:25:14 Atlanta 10.0.0.1 62505 Texas 10.0.0.26 80 HTTP UDP 2178 +2010-12-01 23:25:15 Atlanta 10.0.0.1 49801 Texas 10.0.0.25 80 HTTP UDP 3549 +2010-12-01 23:25:16 Atlanta 10.0.0.1 62132 Texas 10.0.0.27 80 HTTP UDP 3480 +2010-12-01 23:25:17 Atlanta 10.0.0.1 54633 Texas 10.0.0.25 80 HTTP UDP 3198 +2010-12-01 23:25:18 Atlanta 10.0.0.1 58353 Texas 10.0.0.29 80 HTTP UDP 2656 +2010-12-01 23:25:19 Atlanta 10.0.0.1 65316 Texas 10.0.0.30 80 HTTP UDP 4443 +2010-12-01 23:25:20 Atlanta 10.0.0.1 65303 Texas 10.0.0.30 80 HTTP UDP 2788 +2010-12-01 23:25:21 Atlanta 10.0.0.1 60666 Texas 10.0.0.26 80 HTTP UDP 3084 +2010-12-01 23:25:22 Atlanta 10.0.0.1 63378 Texas 10.0.0.30 80 HTTP UDP 3545 +2010-12-01 23:25:23 Atlanta 10.0.0.1 57620 Texas 10.0.0.27 80 HTTP UDP 3009 +2010-12-01 23:25:24 Atlanta 10.0.0.1 64705 Texas 10.0.0.27 80 HTTP UDP 4097 +2010-12-01 23:25:25 Atlanta 10.0.0.1 63447 Texas 10.0.0.30 80 HTTP UDP 3628 +2010-12-01 23:25:26 Atlanta 10.0.0.1 60675 Texas 10.0.0.29 80 HTTP UDP 3174 +2010-12-01 23:25:27 Atlanta 10.0.0.1 64533 Texas 10.0.0.26 80 HTTP UDP 3873 +2010-12-01 23:25:28 Atlanta 10.0.0.1 64657 Texas 10.0.0.25 80 HTTP UDP 4162 +2010-12-01 23:25:29 Atlanta 10.0.0.1 61747 Texas 10.0.0.26 80 HTTP UDP 5345 +2010-12-01 23:25:30 Atlanta 10.0.0.1 51004 Texas 10.0.0.29 80 HTTP UDP 3107 +2010-12-01 23:25:31 Atlanta 10.0.0.1 56437 Texas 10.0.0.29 80 HTTP UDP 3096 +2010-12-01 23:25:32 Atlanta 10.0.0.1 54318 Texas 10.0.0.29 80 HTTP UDP 3955 +2010-12-01 23:25:33 Atlanta 10.0.0.1 57496 Texas 10.0.0.28 80 HTTP UDP 2162 +2010-12-01 23:25:34 Atlanta 10.0.0.1 56481 Texas 10.0.0.27 80 HTTP UDP 4411 +2010-12-01 23:25:35 Atlanta 10.0.0.1 62981 Texas 10.0.0.27 80 HTTP UDP 2526 +2010-12-01 23:25:36 Atlanta 10.0.0.1 52037 Texas 10.0.0.26 80 HTTP UDP 4945 +2010-12-01 23:25:37 Atlanta 10.0.0.1 65391 Texas 10.0.0.30 80 HTTP UDP 5963 +2010-12-01 23:25:38 Atlanta 10.0.0.1 64659 Texas 10.0.0.25 80 HTTP UDP 3936 +2010-12-01 23:25:39 Atlanta 10.0.0.1 49747 Texas 10.0.0.27 80 HTTP UDP 4008 +2010-12-01 23:25:40 Atlanta 10.0.0.1 59157 Texas 10.0.0.26 80 HTTP UDP 3421 +2010-12-01 23:25:41 Atlanta 10.0.0.1 57469 Texas 10.0.0.27 80 HTTP UDP 3230 +2010-12-01 23:25:42 Atlanta 10.0.0.1 49375 Texas 10.0.0.29 80 HTTP UDP 3486 +2010-12-01 23:25:43 Atlanta 10.0.0.1 64263 Texas 10.0.0.29 80 HTTP UDP 5693 +2010-12-01 23:25:44 Atlanta 10.0.0.1 57214 Texas 10.0.0.28 80 HTTP UDP 3893 +2010-12-01 23:25:45 Atlanta 10.0.0.1 63329 Texas 10.0.0.28 80 HTTP UDP 4855 +2010-12-01 23:25:46 Atlanta 10.0.0.1 64033 Texas 10.0.0.27 80 HTTP UDP 2168 +2010-12-01 23:25:47 Atlanta 10.0.0.1 51836 Texas 10.0.0.28 80 HTTP UDP 4411 +2010-12-01 23:25:48 Atlanta 10.0.0.1 57488 Texas 10.0.0.30 80 HTTP UDP 4449 +2010-12-01 23:25:49 Atlanta 10.0.0.1 58476 Texas 10.0.0.27 80 HTTP UDP 5016 +2010-12-01 23:25:50 Atlanta 10.0.0.1 62639 Texas 10.0.0.27 80 HTTP UDP 5221 +2010-12-01 23:25:51 Atlanta 10.0.0.1 65071 Texas 10.0.0.26 80 HTTP UDP 2537 +2010-12-01 23:25:52 Atlanta 10.0.0.1 52733 Texas 10.0.0.27 80 HTTP UDP 5309 +2010-12-01 23:25:53 Atlanta 10.0.0.1 55191 Texas 10.0.0.26 80 HTTP UDP 4936 +2010-12-01 23:25:54 Atlanta 10.0.0.1 60407 Texas 10.0.0.29 80 HTTP UDP 3806 +2010-12-01 23:25:55 Atlanta 10.0.0.1 56702 Texas 10.0.0.28 80 HTTP UDP 3285 +2010-12-01 23:25:56 Atlanta 10.0.0.1 58369 Texas 10.0.0.26 80 HTTP UDP 2479 +2010-12-01 23:25:57 Atlanta 10.0.0.1 61455 Texas 10.0.0.28 80 HTTP UDP 5047 +2010-12-01 23:25:58 Atlanta 10.0.0.1 54872 Texas 10.0.0.26 80 HTTP UDP 2746 +2010-12-01 23:25:59 Atlanta 10.0.0.1 52497 Texas 10.0.0.26 80 HTTP UDP 3223 +2010-12-01 23:26:00 Atlanta 10.0.0.1 63231 Texas 10.0.0.30 80 HTTP UDP 4683 +2010-12-01 23:26:01 Atlanta 10.0.0.1 61181 Texas 10.0.0.28 80 HTTP UDP 2688 +2010-12-01 23:26:02 Atlanta 10.0.0.1 50877 Texas 10.0.0.27 80 HTTP UDP 3908 +2010-12-01 23:26:03 Atlanta 10.0.0.1 56308 Texas 10.0.0.27 80 HTTP UDP 3285 +2010-12-01 23:26:04 Atlanta 10.0.0.1 64666 Texas 10.0.0.30 80 HTTP UDP 2611 +2010-12-01 23:26:05 Atlanta 10.0.0.1 51053 Texas 10.0.0.27 80 HTTP UDP 4461 +2010-12-01 23:26:06 Atlanta 10.0.0.1 65095 Texas 10.0.0.29 80 HTTP UDP 3806 +2010-12-01 23:26:07 Atlanta 10.0.0.1 49340 Texas 10.0.0.25 80 HTTP UDP 3067 +2010-12-01 23:26:08 Atlanta 10.0.0.1 59015 Texas 10.0.0.25 80 HTTP UDP 5002 +2010-12-01 23:26:09 Atlanta 10.0.0.1 63436 Texas 10.0.0.26 80 HTTP UDP 4192 +2010-12-01 23:26:10 Atlanta 10.0.0.1 56513 Texas 10.0.0.25 80 HTTP UDP 4113 +2010-12-01 23:26:11 Atlanta 10.0.0.1 55136 Texas 10.0.0.30 80 HTTP UDP 5501 +2010-12-01 23:26:12 Atlanta 10.0.0.1 53970 Texas 10.0.0.29 80 HTTP UDP 4705 +2010-12-01 23:26:13 Atlanta 10.0.0.1 51585 Texas 10.0.0.29 80 HTTP UDP 2577 +2010-12-01 23:26:14 Atlanta 10.0.0.1 57864 Texas 10.0.0.28 80 HTTP UDP 5543 +2010-12-01 23:26:15 Atlanta 10.0.0.1 59748 Texas 10.0.0.27 80 HTTP UDP 2734 +2010-12-01 23:26:16 Atlanta 10.0.0.1 49924 Texas 10.0.0.27 80 HTTP UDP 4002 +2010-12-01 23:26:17 Atlanta 10.0.0.1 63388 Texas 10.0.0.27 80 HTTP UDP 5952 +2010-12-01 23:26:18 Atlanta 10.0.0.1 60008 Texas 10.0.0.26 80 HTTP UDP 4530 +2010-12-01 23:26:19 Atlanta 10.0.0.1 51006 Texas 10.0.0.28 80 HTTP UDP 5963 +2010-12-01 23:26:20 Atlanta 10.0.0.1 57624 Texas 10.0.0.26 80 HTTP UDP 3114 +2010-12-01 23:26:21 Atlanta 10.0.0.1 50311 Texas 10.0.0.29 80 HTTP UDP 4038 +2010-12-01 23:26:22 Atlanta 10.0.0.1 49737 Texas 10.0.0.28 80 HTTP UDP 5133 +2010-12-01 23:26:23 Atlanta 10.0.0.1 61081 Texas 10.0.0.27 80 HTTP UDP 5696 +2010-12-01 23:26:24 Atlanta 10.0.0.1 64597 Texas 10.0.0.26 80 HTTP UDP 2813 +2010-12-01 23:26:25 Atlanta 10.0.0.1 60575 Texas 10.0.0.25 80 HTTP UDP 5280 +2010-12-01 23:26:26 Atlanta 10.0.0.1 49186 Texas 10.0.0.25 80 HTTP UDP 4173 +2010-12-01 23:26:27 Atlanta 10.0.0.1 58834 Texas 10.0.0.29 80 HTTP UDP 3761 +2010-12-01 23:26:28 Atlanta 10.0.0.1 56327 Texas 10.0.0.30 80 HTTP UDP 3601 +2010-12-01 23:26:29 Atlanta 10.0.0.1 62021 Texas 10.0.0.27 80 HTTP UDP 5894 +2010-12-01 23:26:30 Atlanta 10.0.0.1 61101 Texas 10.0.0.28 80 HTTP UDP 3896 +2010-12-01 23:26:31 Atlanta 10.0.0.1 52868 Texas 10.0.0.28 80 HTTP UDP 5232 +2010-12-01 23:26:32 Atlanta 10.0.0.1 49233 Texas 10.0.0.29 80 HTTP UDP 2088 +2010-12-01 23:26:33 Atlanta 10.0.0.1 57561 Texas 10.0.0.28 80 HTTP UDP 3079 +2010-12-01 23:26:34 Atlanta 10.0.0.1 62805 Texas 10.0.0.25 80 HTTP UDP 5029 +2010-12-01 23:26:35 Atlanta 10.0.0.1 55929 Texas 10.0.0.25 80 HTTP UDP 2523 +2010-12-01 23:26:36 Atlanta 10.0.0.1 56939 Texas 10.0.0.26 80 HTTP UDP 5652 +2010-12-01 23:26:37 Atlanta 10.0.0.1 53256 Texas 10.0.0.29 80 HTTP UDP 4899 +2010-12-01 23:26:38 Atlanta 10.0.0.1 64245 Texas 10.0.0.28 80 HTTP UDP 4229 +2010-12-01 23:26:39 Atlanta 10.0.0.1 59070 Texas 10.0.0.25 80 HTTP UDP 2860 +2010-12-01 23:26:40 Atlanta 10.0.0.1 54203 Texas 10.0.0.29 80 HTTP UDP 2571 +2010-12-01 23:26:41 Atlanta 10.0.0.1 57772 Texas 10.0.0.25 80 HTTP UDP 3990 +2010-12-01 23:26:42 Atlanta 10.0.0.1 57298 Texas 10.0.0.28 80 HTTP UDP 5019 +2010-12-01 23:26:43 Atlanta 10.0.0.1 53570 Texas 10.0.0.28 80 HTTP UDP 3581 +2010-12-01 23:26:44 Atlanta 10.0.0.1 61674 Texas 10.0.0.29 80 HTTP UDP 5232 +2010-12-01 23:26:45 Atlanta 10.0.0.1 63360 Texas 10.0.0.25 80 HTTP UDP 5163 +2010-12-01 23:26:46 Atlanta 10.0.0.1 59827 Texas 10.0.0.28 80 HTTP UDP 3122 +2010-12-01 23:26:47 Atlanta 10.0.0.1 63460 Texas 10.0.0.30 80 HTTP UDP 3421 +2010-12-01 23:26:48 Atlanta 10.0.0.1 63096 Texas 10.0.0.30 80 HTTP UDP 5888 +2010-12-01 23:26:49 Atlanta 10.0.0.1 57500 Texas 10.0.0.27 80 HTTP UDP 2082 +2010-12-01 23:26:50 Atlanta 10.0.0.1 64533 Texas 10.0.0.30 80 HTTP UDP 5867 +2010-12-01 23:26:51 Atlanta 10.0.0.1 51854 Texas 10.0.0.30 80 HTTP UDP 4417 +2010-12-01 23:26:52 Atlanta 10.0.0.1 58266 Texas 10.0.0.26 80 HTTP UDP 2577 +2010-12-01 23:26:53 Atlanta 10.0.0.1 62127 Texas 10.0.0.28 80 HTTP UDP 3538 +2010-12-01 23:26:54 Atlanta 10.0.0.1 56252 Texas 10.0.0.25 80 HTTP UDP 4538 +2010-12-01 23:26:55 Atlanta 10.0.0.1 59817 Texas 10.0.0.28 80 HTTP UDP 5347 +2010-12-01 23:26:56 Atlanta 10.0.0.1 63220 Texas 10.0.0.30 80 HTTP UDP 3872 +2010-12-01 23:26:57 Atlanta 10.0.0.1 49788 Texas 10.0.0.26 80 HTTP UDP 5015 +2010-12-01 23:26:58 Atlanta 10.0.0.1 62532 Texas 10.0.0.27 80 HTTP UDP 5435 +2010-12-01 23:26:59 Atlanta 10.0.0.1 64573 Texas 10.0.0.26 80 HTTP UDP 5435 +2010-12-01 23:27:00 Atlanta 10.0.0.1 54120 Texas 10.0.0.26 80 HTTP UDP 3486 +2010-12-01 23:27:01 Atlanta 10.0.0.1 60165 Texas 10.0.0.29 80 HTTP UDP 5938 +2010-12-01 23:27:02 Atlanta 10.0.0.1 59225 Texas 10.0.0.27 80 HTTP UDP 4379 +2010-12-01 23:27:03 Atlanta 10.0.0.1 49782 Texas 10.0.0.25 80 HTTP UDP 5243 +2010-12-01 23:27:04 Atlanta 10.0.0.1 61850 Texas 10.0.0.29 80 HTTP UDP 5163 +2010-12-01 23:27:05 Atlanta 10.0.0.1 58101 Texas 10.0.0.28 80 HTTP UDP 4920 +2010-12-01 23:27:06 Atlanta 10.0.0.1 62196 Texas 10.0.0.27 80 HTTP UDP 4549 +2010-12-01 23:27:07 Atlanta 10.0.0.1 53449 Texas 10.0.0.26 80 HTTP UDP 2693 +2010-12-01 23:27:08 Atlanta 10.0.0.1 53460 Texas 10.0.0.25 80 HTTP UDP 3979 +2010-12-01 23:27:09 Atlanta 10.0.0.1 65366 Texas 10.0.0.28 80 HTTP UDP 2273 +2010-12-01 23:27:10 Atlanta 10.0.0.1 50737 Texas 10.0.0.27 80 HTTP UDP 5341 +2010-12-01 23:27:11 Atlanta 10.0.0.1 61694 Texas 10.0.0.27 80 HTTP UDP 3704 +2010-12-01 23:27:12 Atlanta 10.0.0.1 57416 Texas 10.0.0.30 80 HTTP UDP 4374 +2010-12-01 23:27:13 Atlanta 10.0.0.1 63343 Texas 10.0.0.25 80 HTTP UDP 4424 +2010-12-01 23:27:14 Atlanta 10.0.0.1 57512 Texas 10.0.0.25 80 HTTP UDP 3899 +2010-12-01 23:27:15 Atlanta 10.0.0.1 50852 Texas 10.0.0.28 80 HTTP UDP 5907 +2010-12-01 23:27:16 Atlanta 10.0.0.1 59870 Texas 10.0.0.25 80 HTTP UDP 2882 +2010-12-01 23:27:17 Atlanta 10.0.0.1 64053 Texas 10.0.0.26 80 HTTP UDP 3398 +2010-12-01 23:27:18 Atlanta 10.0.0.1 61264 Texas 10.0.0.29 80 HTTP UDP 5072 +2010-12-01 23:27:19 Atlanta 10.0.0.1 52961 Texas 10.0.0.28 80 HTTP UDP 2788 +2010-12-01 23:27:20 Atlanta 10.0.0.1 56466 Texas 10.0.0.30 80 HTTP UDP 5280 +2010-12-01 23:27:21 Atlanta 10.0.0.1 59263 Texas 10.0.0.28 80 HTTP UDP 5258 +2010-12-01 23:27:22 Atlanta 10.0.0.1 50371 Texas 10.0.0.29 80 HTTP UDP 5899 +2010-12-01 23:27:23 Atlanta 10.0.0.1 56622 Texas 10.0.0.30 80 HTTP UDP 5256 +2010-12-01 23:27:24 Atlanta 10.0.0.1 51567 Texas 10.0.0.29 80 HTTP UDP 3025 +2010-12-01 23:27:25 Atlanta 10.0.0.1 56337 Texas 10.0.0.28 80 HTTP UDP 4195 +2010-12-01 23:27:26 Atlanta 10.0.0.1 50240 Texas 10.0.0.27 80 HTTP UDP 3673 +2010-12-01 23:27:27 Atlanta 10.0.0.1 54167 Texas 10.0.0.30 80 HTTP UDP 3585 +2010-12-01 23:27:28 Atlanta 10.0.0.1 60048 Texas 10.0.0.26 80 HTTP UDP 4633 +2010-12-01 23:27:29 Atlanta 10.0.0.1 61287 Texas 10.0.0.27 80 HTTP UDP 3469 +2010-12-01 23:27:30 Atlanta 10.0.0.1 58410 Texas 10.0.0.27 80 HTTP UDP 4827 +2010-12-01 23:27:31 Atlanta 10.0.0.1 63867 Texas 10.0.0.25 80 HTTP UDP 2029 +2010-12-01 23:27:32 Atlanta 10.0.0.1 54892 Texas 10.0.0.29 80 HTTP UDP 5675 +2010-12-01 23:27:33 Atlanta 10.0.0.1 52531 Texas 10.0.0.25 80 HTTP UDP 3334 +2010-12-01 23:27:34 Atlanta 10.0.0.1 53798 Texas 10.0.0.26 80 HTTP UDP 4393 +2010-12-01 23:27:35 Atlanta 10.0.0.1 49244 Texas 10.0.0.25 80 HTTP UDP 5258 +2010-12-01 23:27:36 Atlanta 10.0.0.1 49312 Texas 10.0.0.28 80 HTTP UDP 2663 +2010-12-01 23:27:37 Atlanta 10.0.0.1 50144 Texas 10.0.0.28 80 HTTP UDP 3147 +2010-12-01 23:27:38 Atlanta 10.0.0.1 64533 Texas 10.0.0.30 80 HTTP UDP 3995 +2010-12-01 23:27:39 Atlanta 10.0.0.1 54711 Texas 10.0.0.25 80 HTTP UDP 4931 +2010-12-01 23:27:40 Atlanta 10.0.0.1 49895 Texas 10.0.0.28 80 HTTP UDP 3571 +2010-12-01 23:27:41 Atlanta 10.0.0.1 56485 Texas 10.0.0.27 80 HTTP UDP 5243 +2010-12-01 23:27:42 Atlanta 10.0.0.1 58715 Texas 10.0.0.27 80 HTTP UDP 3861 +2010-12-01 23:27:43 Atlanta 10.0.0.1 62065 Texas 10.0.0.30 80 HTTP UDP 5348 +2010-12-01 23:27:44 Atlanta 10.0.0.1 52054 Texas 10.0.0.28 80 HTTP UDP 3793 +2010-12-01 23:27:45 Atlanta 10.0.0.1 50424 Texas 10.0.0.28 80 HTTP UDP 3416 +2010-12-01 23:27:46 Atlanta 10.0.0.1 57843 Texas 10.0.0.28 80 HTTP UDP 5035 +2010-12-01 23:27:47 Atlanta 10.0.0.1 54749 Texas 10.0.0.28 80 HTTP UDP 2308 +2010-12-01 23:27:48 Atlanta 10.0.0.1 65316 Texas 10.0.0.26 80 HTTP UDP 2985 +2010-12-01 23:27:49 Atlanta 10.0.0.1 56059 Texas 10.0.0.27 80 HTTP UDP 2442 +2010-12-01 23:27:50 Atlanta 10.0.0.1 51486 Texas 10.0.0.28 80 HTTP UDP 2442 +2010-12-01 23:27:51 Atlanta 10.0.0.1 51996 Texas 10.0.0.30 80 HTTP UDP 5106 +2010-12-01 23:27:52 Atlanta 10.0.0.1 55528 Texas 10.0.0.29 80 HTTP UDP 4744 +2010-12-01 23:27:53 Atlanta 10.0.0.1 53449 Texas 10.0.0.29 80 HTTP UDP 4683 +2010-12-01 23:27:54 Atlanta 10.0.0.1 61761 Texas 10.0.0.27 80 HTTP UDP 5057 +2010-12-01 23:27:55 Atlanta 10.0.0.1 62539 Texas 10.0.0.28 80 HTTP UDP 3918 +2010-12-01 23:27:56 Atlanta 10.0.0.1 60196 Texas 10.0.0.26 80 HTTP UDP 3601 +2010-12-01 23:27:57 Atlanta 10.0.0.1 64113 Texas 10.0.0.27 80 HTTP UDP 3889 +2010-12-01 23:27:58 Atlanta 10.0.0.1 55981 Texas 10.0.0.29 80 HTTP UDP 5211 +2010-12-01 23:27:59 Atlanta 10.0.0.1 53375 Texas 10.0.0.27 80 HTTP UDP 3402 +2010-12-01 23:28:00 Atlanta 10.0.0.1 49852 Texas 10.0.0.26 80 HTTP UDP 2308 +2010-12-01 23:28:01 Atlanta 10.0.0.1 64835 Texas 10.0.0.29 80 HTTP UDP 3104 +2010-12-01 23:28:02 Atlanta 10.0.0.1 52237 Texas 10.0.0.29 80 HTTP UDP 5501 +2010-12-01 23:28:03 Atlanta 10.0.0.1 51924 Texas 10.0.0.28 80 HTTP UDP 4804 +2010-12-01 23:28:04 Atlanta 10.0.0.1 59869 Texas 10.0.0.29 80 HTTP UDP 3663 +2010-12-01 23:28:05 Atlanta 10.0.0.1 59931 Texas 10.0.0.27 80 HTTP UDP 4293 +2010-12-01 23:28:06 Atlanta 10.0.0.1 62156 Texas 10.0.0.29 80 HTTP UDP 4324 +2010-12-01 23:28:07 Atlanta 10.0.0.1 57237 Texas 10.0.0.29 80 HTTP UDP 5197 +2010-12-01 23:28:08 Atlanta 10.0.0.1 50009 Texas 10.0.0.27 80 HTTP UDP 2674 +2010-12-01 23:28:09 Atlanta 10.0.0.1 58608 Texas 10.0.0.26 80 HTTP UDP 5628 +2010-12-01 23:28:10 Atlanta 10.0.0.1 61007 Texas 10.0.0.25 80 HTTP UDP 3211 +2010-12-01 23:28:11 Atlanta 10.0.0.1 56130 Texas 10.0.0.27 80 HTTP UDP 2329 +2010-12-01 23:28:12 Atlanta 10.0.0.1 49967 Texas 10.0.0.29 80 HTTP UDP 4665 +2010-12-01 23:28:13 Atlanta 10.0.0.1 61872 Texas 10.0.0.27 80 HTTP UDP 5696 +2010-12-01 23:28:14 Atlanta 10.0.0.1 60929 Texas 10.0.0.25 80 HTTP UDP 3450 +2010-12-01 23:28:15 Atlanta 10.0.0.1 58175 Texas 10.0.0.28 80 HTTP UDP 2099 +2010-12-01 23:28:16 Atlanta 10.0.0.1 55805 Texas 10.0.0.26 80 HTTP UDP 2877 +2010-12-01 23:28:17 Atlanta 10.0.0.1 56805 Texas 10.0.0.30 80 HTTP UDP 4164 +2010-12-01 23:28:18 Atlanta 10.0.0.1 59963 Texas 10.0.0.27 80 HTTP UDP 5275 +2010-12-01 23:28:19 Atlanta 10.0.0.1 58555 Texas 10.0.0.27 80 HTTP UDP 2790 +2010-12-01 23:28:20 Atlanta 10.0.0.1 61186 Texas 10.0.0.26 80 HTTP UDP 5950 +2010-12-01 23:28:21 Atlanta 10.0.0.1 49454 Texas 10.0.0.28 80 HTTP UDP 5482 +2010-12-01 23:28:22 Atlanta 10.0.0.1 59157 Texas 10.0.0.25 80 HTTP UDP 2988 +2010-12-01 23:28:23 Atlanta 10.0.0.1 59638 Texas 10.0.0.27 80 HTTP UDP 5217 +2010-12-01 23:28:24 Atlanta 10.0.0.1 56406 Texas 10.0.0.25 80 HTTP UDP 5671 +2010-12-01 23:28:25 Atlanta 10.0.0.1 60075 Texas 10.0.0.26 80 HTTP UDP 4080 +2010-12-01 23:28:26 Atlanta 10.0.0.1 64448 Texas 10.0.0.30 80 HTTP UDP 5545 +2010-12-01 23:28:27 Atlanta 10.0.0.1 53856 Texas 10.0.0.26 80 HTTP UDP 4959 +2010-12-01 23:28:28 Atlanta 10.0.0.1 62962 Texas 10.0.0.26 80 HTTP UDP 4058 +2010-12-01 23:28:29 Atlanta 10.0.0.1 54665 Texas 10.0.0.29 80 HTTP UDP 5850 +2010-12-01 23:28:30 Atlanta 10.0.0.1 58112 Texas 10.0.0.30 80 HTTP UDP 5552 +2010-12-01 23:28:31 Atlanta 10.0.0.1 54459 Texas 10.0.0.27 80 HTTP UDP 4365 +2010-12-01 23:28:32 Atlanta 10.0.0.1 64685 Texas 10.0.0.25 80 HTTP UDP 3872 +2010-12-01 23:28:33 Atlanta 10.0.0.1 55463 Texas 10.0.0.29 80 HTTP UDP 5313 +2010-12-01 23:28:34 Atlanta 10.0.0.1 58369 Texas 10.0.0.29 80 HTTP UDP 4093 +2010-12-01 23:28:35 Atlanta 10.0.0.1 59567 Texas 10.0.0.26 80 HTTP UDP 4356 +2010-12-01 23:28:36 Atlanta 10.0.0.1 62368 Texas 10.0.0.26 80 HTTP UDP 4887 +2010-12-01 23:28:37 Atlanta 10.0.0.1 55823 Texas 10.0.0.30 80 HTTP UDP 3147 +2010-12-01 23:28:38 Atlanta 10.0.0.1 51942 Texas 10.0.0.28 80 HTTP UDP 5231 +2010-12-01 23:28:39 Atlanta 10.0.0.1 57948 Texas 10.0.0.29 80 HTTP UDP 2295 +2010-12-01 23:28:40 Atlanta 10.0.0.1 52859 Texas 10.0.0.28 80 HTTP UDP 4368 +2010-12-01 23:28:41 Atlanta 10.0.0.1 52066 Texas 10.0.0.26 80 HTTP UDP 2221 +2010-12-01 23:28:42 Atlanta 10.0.0.1 54025 Texas 10.0.0.29 80 HTTP UDP 3512 +2010-12-01 23:28:43 Atlanta 10.0.0.1 52735 Texas 10.0.0.28 80 HTTP UDP 3292 +2010-12-01 23:28:44 Atlanta 10.0.0.1 53012 Texas 10.0.0.25 80 HTTP UDP 5377 +2010-12-01 23:28:45 Atlanta 10.0.0.1 65398 Texas 10.0.0.26 80 HTTP UDP 2235 +2010-12-01 23:28:46 Atlanta 10.0.0.1 63712 Texas 10.0.0.29 80 HTTP UDP 3374 +2010-12-01 23:28:47 Atlanta 10.0.0.1 55479 Texas 10.0.0.30 80 HTTP UDP 4293 +2010-12-01 23:28:48 Atlanta 10.0.0.1 58428 Texas 10.0.0.25 80 HTTP UDP 3010 +2010-12-01 23:28:49 Atlanta 10.0.0.1 50098 Texas 10.0.0.28 80 HTTP UDP 5371 +2010-12-01 23:28:50 Atlanta 10.0.0.1 61235 Texas 10.0.0.25 80 HTTP UDP 2579 +2010-12-01 23:28:51 Atlanta 10.0.0.1 63555 Texas 10.0.0.25 80 HTTP UDP 4935 +2010-12-01 23:28:52 Atlanta 10.0.0.1 51433 Texas 10.0.0.25 80 HTTP UDP 4379 +2010-12-01 23:28:53 Atlanta 10.0.0.1 65384 Texas 10.0.0.26 80 HTTP UDP 3028 +2010-12-01 23:28:54 Atlanta 10.0.0.1 63988 Texas 10.0.0.27 80 HTTP UDP 2294 +2010-12-01 23:28:55 Atlanta 10.0.0.1 49212 Texas 10.0.0.29 80 HTTP UDP 4193 +2010-12-01 23:28:56 Atlanta 10.0.0.1 51143 Texas 10.0.0.28 80 HTTP UDP 4368 +2010-12-01 23:28:57 Atlanta 10.0.0.1 65449 Texas 10.0.0.26 80 HTTP UDP 2380 +2010-12-01 23:28:58 Atlanta 10.0.0.1 49427 Texas 10.0.0.27 80 HTTP UDP 3183 +2010-12-01 23:28:59 Atlanta 10.0.0.1 62769 Texas 10.0.0.29 80 HTTP UDP 2024 +2010-12-01 23:29:00 Atlanta 10.0.0.1 53388 Texas 10.0.0.27 80 HTTP UDP 3543 +2010-12-01 23:29:01 Atlanta 10.0.0.1 52199 Texas 10.0.0.29 80 HTTP UDP 3454 +2010-12-01 23:29:02 Atlanta 10.0.0.1 49417 Texas 10.0.0.29 80 HTTP UDP 5096 +2010-12-01 23:29:03 Atlanta 10.0.0.1 61884 Texas 10.0.0.28 80 HTTP UDP 5899 +2010-12-01 23:29:04 Atlanta 10.0.0.1 64210 Texas 10.0.0.26 80 HTTP UDP 2975 +2010-12-01 23:29:05 Atlanta 10.0.0.1 52454 Texas 10.0.0.29 80 HTTP UDP 2007 +2010-12-01 23:29:06 Atlanta 10.0.0.1 65500 Texas 10.0.0.28 80 HTTP UDP 2168 +2010-12-01 23:29:07 Atlanta 10.0.0.1 55732 Texas 10.0.0.29 80 HTTP UDP 3545 +2010-12-01 23:29:08 Atlanta 10.0.0.1 62693 Texas 10.0.0.25 80 HTTP UDP 3736 +2010-12-01 23:29:09 Atlanta 10.0.0.1 59546 Texas 10.0.0.27 80 HTTP UDP 5215 +2010-12-01 23:29:10 Atlanta 10.0.0.1 61244 Texas 10.0.0.28 80 HTTP UDP 3970 +2010-12-01 23:29:11 Atlanta 10.0.0.1 65006 Texas 10.0.0.27 80 HTTP UDP 3070 +2010-12-01 23:29:12 Atlanta 10.0.0.1 59978 Texas 10.0.0.26 80 HTTP UDP 4735 +2010-12-01 23:29:13 Atlanta 10.0.0.1 50694 Texas 10.0.0.28 80 HTTP UDP 5047 +2010-12-01 23:29:14 Atlanta 10.0.0.1 56320 Texas 10.0.0.27 80 HTTP UDP 5712 +2010-12-01 23:29:15 Atlanta 10.0.0.1 50405 Texas 10.0.0.27 80 HTTP UDP 5169 +2010-12-01 23:29:16 Atlanta 10.0.0.1 50072 Texas 10.0.0.29 80 HTTP UDP 2895 +2010-12-01 23:29:17 Atlanta 10.0.0.1 59759 Texas 10.0.0.29 80 HTTP UDP 4867 +2010-12-01 23:29:18 Atlanta 10.0.0.1 61496 Texas 10.0.0.25 80 HTTP UDP 5191 +2010-12-01 23:29:19 Atlanta 10.0.0.1 58823 Texas 10.0.0.29 80 HTTP UDP 3893 +2010-12-01 23:29:20 Atlanta 10.0.0.1 59403 Texas 10.0.0.25 80 HTTP UDP 4921 +2010-12-01 23:29:21 Atlanta 10.0.0.1 65186 Texas 10.0.0.28 80 HTTP UDP 4032 +2010-12-01 23:29:22 Atlanta 10.0.0.1 56886 Texas 10.0.0.25 80 HTTP UDP 4077 +2010-12-01 23:29:23 Atlanta 10.0.0.1 55229 Texas 10.0.0.27 80 HTTP UDP 2050 +2010-12-01 23:29:24 Atlanta 10.0.0.1 60254 Texas 10.0.0.28 80 HTTP UDP 2667 +2010-12-01 23:29:25 Atlanta 10.0.0.1 60032 Texas 10.0.0.29 80 HTTP UDP 3398 +2010-12-01 23:29:26 Atlanta 10.0.0.1 55223 Texas 10.0.0.28 80 HTTP UDP 3628 +2010-12-01 23:29:27 Atlanta 10.0.0.1 61289 Texas 10.0.0.30 80 HTTP UDP 4849 +2010-12-01 23:29:28 Atlanta 10.0.0.1 59765 Texas 10.0.0.28 80 HTTP UDP 5397 +2010-12-01 23:29:29 Atlanta 10.0.0.1 61609 Texas 10.0.0.27 80 HTTP UDP 4502 +2010-12-01 23:29:30 Atlanta 10.0.0.1 55434 Texas 10.0.0.27 80 HTTP UDP 2480 +2010-12-01 23:29:31 Atlanta 10.0.0.1 63772 Texas 10.0.0.25 80 HTTP UDP 3969 +2010-12-01 23:29:32 Atlanta 10.0.0.1 56933 Texas 10.0.0.25 80 HTTP UDP 4176 +2010-12-01 23:29:33 Atlanta 10.0.0.1 56066 Texas 10.0.0.28 80 HTTP UDP 4502 +2010-12-01 23:29:34 Atlanta 10.0.0.1 61350 Texas 10.0.0.25 80 HTTP UDP 2391 +2010-12-01 23:29:35 Atlanta 10.0.0.1 54260 Texas 10.0.0.28 80 HTTP UDP 2217 +2010-12-01 23:29:36 Atlanta 10.0.0.1 62691 Texas 10.0.0.30 80 HTTP UDP 4969 +2010-12-01 23:29:37 Atlanta 10.0.0.1 60110 Texas 10.0.0.26 80 HTTP UDP 4258 +2010-12-01 23:29:38 Atlanta 10.0.0.1 52167 Texas 10.0.0.26 80 HTTP UDP 2154 +2010-12-01 23:29:39 Atlanta 10.0.0.1 62607 Texas 10.0.0.25 80 HTTP UDP 2906 +2010-12-01 23:29:40 Atlanta 10.0.0.1 62234 Texas 10.0.0.30 80 HTTP UDP 2422 +2010-12-01 23:29:41 Atlanta 10.0.0.1 61167 Texas 10.0.0.29 80 HTTP UDP 3216 +2010-12-01 23:29:42 Atlanta 10.0.0.1 50691 Texas 10.0.0.27 80 HTTP UDP 5017 +2010-12-01 23:29:43 Atlanta 10.0.0.1 55728 Texas 10.0.0.27 80 HTTP UDP 5888 +2010-12-01 23:29:44 Atlanta 10.0.0.1 61374 Texas 10.0.0.25 80 HTTP UDP 2813 +2010-12-01 23:29:45 Atlanta 10.0.0.1 49317 Texas 10.0.0.29 80 HTTP UDP 2595 +2010-12-01 23:29:46 Atlanta 10.0.0.1 56488 Texas 10.0.0.25 80 HTTP UDP 4502 +2010-12-01 23:29:47 Atlanta 10.0.0.1 64805 Texas 10.0.0.26 80 HTTP UDP 2525 +2010-12-01 23:29:48 Atlanta 10.0.0.1 53322 Texas 10.0.0.28 80 HTTP UDP 2563 +2010-12-01 23:29:49 Atlanta 10.0.0.1 53427 Texas 10.0.0.28 80 HTTP UDP 4931 +2010-12-01 23:29:50 Atlanta 10.0.0.1 59463 Texas 10.0.0.30 80 HTTP UDP 3947 +2010-12-01 23:29:51 Atlanta 10.0.0.1 58781 Texas 10.0.0.25 80 HTTP UDP 3032 +2010-12-01 23:29:52 Atlanta 10.0.0.1 51051 Texas 10.0.0.25 80 HTTP UDP 4977 +2010-12-01 23:29:53 Atlanta 10.0.0.1 58986 Texas 10.0.0.25 80 HTTP UDP 5309 +2010-12-01 23:29:54 Atlanta 10.0.0.1 62541 Texas 10.0.0.27 80 HTTP UDP 4683 +2010-12-01 23:29:55 Atlanta 10.0.0.1 63007 Texas 10.0.0.26 80 HTTP UDP 3131 +2010-12-01 23:29:56 Atlanta 10.0.0.1 51363 Texas 10.0.0.25 80 HTTP UDP 3182 +2010-12-01 23:29:57 Atlanta 10.0.0.1 62739 Texas 10.0.0.25 80 HTTP UDP 4506 +2010-12-01 23:29:58 Atlanta 10.0.0.1 64170 Texas 10.0.0.25 80 HTTP UDP 4627 +2010-12-01 23:29:59 Atlanta 10.0.0.1 50642 Texas 10.0.0.25 80 HTTP UDP 4279 +2010-12-01 23:30:00 Atlanta 10.0.0.1 58075 Texas 10.0.0.25 80 HTTP UDP 3935 +2010-12-01 23:30:01 Atlanta 10.0.0.1 61270 Texas 10.0.0.30 80 HTTP UDP 5135 +2010-12-01 23:30:02 Atlanta 10.0.0.1 56105 Texas 10.0.0.29 80 HTTP UDP 3934 +2010-12-01 23:30:03 Atlanta 10.0.0.1 55041 Texas 10.0.0.27 80 HTTP UDP 3607 +2010-12-01 23:30:04 Atlanta 10.0.0.1 57914 Texas 10.0.0.30 80 HTTP UDP 2319 +2010-12-01 23:30:05 Atlanta 10.0.0.1 57052 Texas 10.0.0.25 80 HTTP UDP 3357 +2010-12-01 23:30:06 Atlanta 10.0.0.1 51021 Texas 10.0.0.28 80 HTTP UDP 5925 +2010-12-01 23:30:07 Atlanta 10.0.0.1 49830 Texas 10.0.0.25 80 HTTP UDP 4620 +2010-12-01 23:30:08 Atlanta 10.0.0.1 52656 Texas 10.0.0.27 80 HTTP UDP 2681 +2010-12-01 23:30:09 Atlanta 10.0.0.1 59501 Texas 10.0.0.29 80 HTTP UDP 5948 +2010-12-01 23:30:10 Atlanta 10.0.0.1 61289 Texas 10.0.0.28 80 HTTP UDP 5039 +2010-12-01 23:30:11 Atlanta 10.0.0.1 52185 Texas 10.0.0.28 80 HTTP UDP 2924 +2010-12-01 23:30:12 Atlanta 10.0.0.1 62838 Texas 10.0.0.26 80 HTTP UDP 3634 +2010-12-01 23:30:13 Atlanta 10.0.0.1 56720 Texas 10.0.0.29 80 HTTP UDP 2576 +2010-12-01 23:30:14 Atlanta 10.0.0.1 62150 Texas 10.0.0.27 80 HTTP UDP 2024 +2010-12-01 23:30:15 Atlanta 10.0.0.1 53535 Texas 10.0.0.29 80 HTTP UDP 5506 +2010-12-01 23:30:16 Atlanta 10.0.0.1 61027 Texas 10.0.0.29 80 HTTP UDP 4171 +2010-12-01 23:30:17 Atlanta 10.0.0.1 50177 Texas 10.0.0.30 80 HTTP UDP 4771 +2010-12-01 23:30:18 Atlanta 10.0.0.1 59905 Texas 10.0.0.28 80 HTTP UDP 3647 +2010-12-01 23:30:19 Atlanta 10.0.0.1 49277 Texas 10.0.0.29 80 HTTP UDP 5668 +2010-12-01 23:30:20 Atlanta 10.0.0.1 49511 Texas 10.0.0.30 80 HTTP UDP 3099 +2010-12-01 23:30:21 Atlanta 10.0.0.1 52262 Texas 10.0.0.29 80 HTTP UDP 3419 +2010-12-01 23:30:22 Atlanta 10.0.0.1 53419 Texas 10.0.0.27 80 HTTP UDP 5462 +2010-12-01 23:30:23 Atlanta 10.0.0.1 55523 Texas 10.0.0.25 80 HTTP UDP 5868 +2010-12-01 23:30:24 Atlanta 10.0.0.1 63666 Texas 10.0.0.29 80 HTTP UDP 4088 +2010-12-01 23:30:25 Atlanta 10.0.0.1 64579 Texas 10.0.0.30 80 HTTP UDP 2746 +2010-12-01 23:30:26 Atlanta 10.0.0.1 57317 Texas 10.0.0.28 80 HTTP UDP 5232 +2010-12-01 23:30:27 Atlanta 10.0.0.1 60054 Texas 10.0.0.25 80 HTTP UDP 4153 +2010-12-01 23:30:28 Atlanta 10.0.0.1 64366 Texas 10.0.0.30 80 HTTP UDP 3486 +2010-12-01 23:30:29 Atlanta 10.0.0.1 61542 Texas 10.0.0.26 80 HTTP UDP 3758 +2010-12-01 23:30:30 Atlanta 10.0.0.1 62349 Texas 10.0.0.27 80 HTTP UDP 4273 +2010-12-01 23:30:31 Atlanta 10.0.0.1 65128 Texas 10.0.0.25 80 HTTP UDP 3019 +2010-12-01 23:30:32 Atlanta 10.0.0.1 64219 Texas 10.0.0.27 80 HTTP UDP 4179 +2010-12-01 23:30:33 Atlanta 10.0.0.1 64710 Texas 10.0.0.29 80 HTTP UDP 2480 +2010-12-01 23:30:34 Atlanta 10.0.0.1 51983 Texas 10.0.0.30 80 HTTP UDP 3196 +2010-12-01 23:30:35 Atlanta 10.0.0.1 59205 Texas 10.0.0.30 80 HTTP UDP 5710 +2010-12-01 23:30:36 Atlanta 10.0.0.1 56760 Texas 10.0.0.28 80 HTTP UDP 5034 +2010-12-01 23:30:37 Atlanta 10.0.0.1 58176 Texas 10.0.0.29 80 HTTP UDP 4379 +2010-12-01 23:30:38 Atlanta 10.0.0.1 55173 Texas 10.0.0.29 80 HTTP UDP 4179 +2010-12-01 23:30:39 Atlanta 10.0.0.1 52453 Texas 10.0.0.25 80 HTTP UDP 5204 +2010-12-01 23:30:40 Atlanta 10.0.0.1 56275 Texas 10.0.0.25 80 HTTP UDP 3625 +2010-12-01 23:30:41 Atlanta 10.0.0.1 56017 Texas 10.0.0.28 80 HTTP UDP 3345 +2010-12-01 23:30:42 Atlanta 10.0.0.1 55324 Texas 10.0.0.25 80 HTTP UDP 2079 +2010-12-01 23:30:43 Atlanta 10.0.0.1 50129 Texas 10.0.0.28 80 HTTP UDP 4414 +2010-12-01 23:30:44 Atlanta 10.0.0.1 49211 Texas 10.0.0.29 80 HTTP UDP 3592 +2010-12-01 23:30:45 Atlanta 10.0.0.1 58964 Texas 10.0.0.30 80 HTTP UDP 2660 +2010-12-01 23:30:46 Atlanta 10.0.0.1 64411 Texas 10.0.0.26 80 HTTP UDP 3317 +2010-12-01 23:30:47 Atlanta 10.0.0.1 64002 Texas 10.0.0.27 80 HTTP UDP 5258 +2010-12-01 23:30:48 Atlanta 10.0.0.1 50204 Texas 10.0.0.28 80 HTTP UDP 5506 +2010-12-01 23:30:49 Atlanta 10.0.0.1 53898 Texas 10.0.0.28 80 HTTP UDP 4934 +2010-12-01 23:30:50 Atlanta 10.0.0.1 56007 Texas 10.0.0.27 80 HTTP UDP 4638 +2010-12-01 23:30:51 Atlanta 10.0.0.1 54087 Texas 10.0.0.27 80 HTTP UDP 5072 +2010-12-01 23:30:52 Atlanta 10.0.0.1 56950 Texas 10.0.0.30 80 HTTP UDP 4021 +2010-12-01 23:30:53 Atlanta 10.0.0.1 61443 Texas 10.0.0.30 80 HTTP UDP 4670 +2010-12-01 23:30:54 Atlanta 10.0.0.1 53282 Texas 10.0.0.29 80 HTTP UDP 4135 +2010-12-01 23:30:55 Atlanta 10.0.0.1 53630 Texas 10.0.0.25 80 HTTP UDP 2326 +2010-12-01 23:30:56 Atlanta 10.0.0.1 53440 Texas 10.0.0.29 80 HTTP UDP 3326 +2010-12-01 23:30:57 Atlanta 10.0.0.1 49253 Texas 10.0.0.29 80 HTTP UDP 2479 +2010-12-01 23:30:58 Atlanta 10.0.0.1 56638 Texas 10.0.0.26 80 HTTP UDP 2319 +2010-12-01 23:30:59 Atlanta 10.0.0.1 57495 Texas 10.0.0.28 80 HTTP UDP 5775 +2010-12-01 23:31:00 Atlanta 10.0.0.1 64679 Texas 10.0.0.27 80 HTTP UDP 4252 +2010-12-01 23:31:01 Atlanta 10.0.0.1 49731 Texas 10.0.0.30 80 HTTP UDP 3935 +2010-12-01 23:31:02 Atlanta 10.0.0.1 54260 Texas 10.0.0.27 80 HTTP UDP 4879 +2010-12-01 23:31:03 Atlanta 10.0.0.1 62150 Texas 10.0.0.28 80 HTTP UDP 4165 +2010-12-01 23:31:04 Atlanta 10.0.0.1 53895 Texas 10.0.0.29 80 HTTP UDP 5501 +2010-12-01 23:31:05 Atlanta 10.0.0.1 55269 Texas 10.0.0.30 80 HTTP UDP 3996 +2010-12-01 23:31:06 Atlanta 10.0.0.1 61212 Texas 10.0.0.30 80 HTTP UDP 5072 +2010-12-01 23:31:07 Atlanta 10.0.0.1 50396 Texas 10.0.0.26 80 HTTP UDP 3009 +2010-12-01 23:31:08 Atlanta 10.0.0.1 60828 Texas 10.0.0.25 80 HTTP UDP 4977 +2010-12-01 23:31:09 Atlanta 10.0.0.1 56676 Texas 10.0.0.29 80 HTTP UDP 4902 +2010-12-01 23:31:10 Atlanta 10.0.0.1 53239 Texas 10.0.0.28 80 HTTP UDP 5888 +2010-12-01 23:31:11 Atlanta 10.0.0.1 50064 Texas 10.0.0.28 80 HTTP UDP 2997 +2010-12-01 23:31:12 Atlanta 10.0.0.1 52475 Texas 10.0.0.25 80 HTTP UDP 2263 +2010-12-01 23:31:13 Atlanta 10.0.0.1 63272 Texas 10.0.0.30 80 HTTP UDP 4327 +2010-12-01 23:31:14 Atlanta 10.0.0.1 63265 Texas 10.0.0.26 80 HTTP UDP 2378 +2010-12-01 23:31:15 Atlanta 10.0.0.1 52228 Texas 10.0.0.27 80 HTTP UDP 2231 +2010-12-01 23:31:16 Atlanta 10.0.0.1 60745 Texas 10.0.0.27 80 HTTP UDP 2734 +2010-12-01 23:31:17 Atlanta 10.0.0.1 64247 Texas 10.0.0.29 80 HTTP UDP 2221 +2010-12-01 23:31:18 Atlanta 10.0.0.1 56769 Texas 10.0.0.29 80 HTTP UDP 5925 +2010-12-01 23:31:19 Atlanta 10.0.0.1 63204 Texas 10.0.0.26 80 HTTP UDP 5127 +2010-12-01 23:31:20 Atlanta 10.0.0.1 64412 Texas 10.0.0.26 80 HTTP UDP 5014 +2010-12-01 23:31:21 Atlanta 10.0.0.1 54571 Texas 10.0.0.28 80 HTTP UDP 2272 +2010-12-01 23:31:22 Atlanta 10.0.0.1 50436 Texas 10.0.0.26 80 HTTP UDP 4501 +2010-12-01 23:31:23 Atlanta 10.0.0.1 49449 Texas 10.0.0.25 80 HTTP UDP 3657 +2010-12-01 23:31:24 Atlanta 10.0.0.1 57005 Texas 10.0.0.27 80 HTTP UDP 5309 +2010-12-01 23:31:25 Atlanta 10.0.0.1 55088 Texas 10.0.0.27 80 HTTP UDP 5565 +2010-12-01 23:31:26 Atlanta 10.0.0.1 51081 Texas 10.0.0.27 80 HTTP UDP 5634 +2010-12-01 23:31:27 Atlanta 10.0.0.1 54064 Texas 10.0.0.28 80 HTTP UDP 2172 +2010-12-01 23:31:28 Atlanta 10.0.0.1 59361 Texas 10.0.0.25 80 HTTP UDP 4647 +2010-12-01 23:31:29 Atlanta 10.0.0.1 50968 Texas 10.0.0.30 80 HTTP UDP 4496 +2010-12-01 23:31:30 Atlanta 10.0.0.1 61794 Texas 10.0.0.25 80 HTTP UDP 3149 +2010-12-01 23:31:31 Atlanta 10.0.0.1 62199 Texas 10.0.0.29 80 HTTP UDP 4414 +2010-12-01 23:31:32 Atlanta 10.0.0.1 59721 Texas 10.0.0.25 80 HTTP UDP 4849 +2010-12-01 23:31:33 Atlanta 10.0.0.1 49387 Texas 10.0.0.25 80 HTTP UDP 3135 +2010-12-01 23:31:34 Atlanta 10.0.0.1 64271 Texas 10.0.0.29 80 HTTP UDP 5392 +2010-12-01 23:31:35 Atlanta 10.0.0.1 65437 Texas 10.0.0.29 80 HTTP UDP 4969 +2010-12-01 23:31:36 Atlanta 10.0.0.1 50250 Texas 10.0.0.29 80 HTTP UDP 5482 +2010-12-01 23:31:37 Atlanta 10.0.0.1 62278 Texas 10.0.0.29 80 HTTP UDP 3013 +2010-12-01 23:31:38 Atlanta 10.0.0.1 50135 Texas 10.0.0.29 80 HTTP UDP 3875 +2010-12-01 23:31:39 Atlanta 10.0.0.1 50279 Texas 10.0.0.26 80 HTTP UDP 3857 +2010-12-01 23:31:40 Atlanta 10.0.0.1 60337 Texas 10.0.0.27 80 HTTP UDP 3723 +2010-12-01 23:31:41 Atlanta 10.0.0.1 63149 Texas 10.0.0.29 80 HTTP UDP 4414 +2010-12-01 23:31:42 Atlanta 10.0.0.1 65217 Texas 10.0.0.25 80 HTTP UDP 5700 +2010-12-01 23:31:43 Atlanta 10.0.0.1 65246 Texas 10.0.0.30 80 HTTP UDP 3990 +2010-12-01 23:31:44 Atlanta 10.0.0.1 53063 Texas 10.0.0.29 80 HTTP UDP 3790 +2010-12-01 23:31:45 Atlanta 10.0.0.1 65128 Texas 10.0.0.30 80 HTTP UDP 3156 +2010-12-01 23:31:46 Atlanta 10.0.0.1 64702 Texas 10.0.0.29 80 HTTP UDP 3171 +2010-12-01 23:31:47 Atlanta 10.0.0.1 51859 Texas 10.0.0.27 80 HTTP UDP 4925 +2010-12-01 23:31:48 Atlanta 10.0.0.1 63618 Texas 10.0.0.25 80 HTTP UDP 2875 +2010-12-01 23:31:49 Atlanta 10.0.0.1 55068 Texas 10.0.0.26 80 HTTP UDP 2891 +2010-12-01 23:31:50 Atlanta 10.0.0.1 60230 Texas 10.0.0.27 80 HTTP UDP 2168 +2010-12-01 23:31:51 Atlanta 10.0.0.1 51118 Texas 10.0.0.28 80 HTTP UDP 5327 +2010-12-01 23:31:52 Atlanta 10.0.0.1 52946 Texas 10.0.0.25 80 HTTP UDP 5340 +2010-12-01 23:31:53 Atlanta 10.0.0.1 50207 Texas 10.0.0.28 80 HTTP UDP 4802 +2010-12-01 23:31:54 Atlanta 10.0.0.1 63395 Texas 10.0.0.27 80 HTTP UDP 5041 +2010-12-01 23:31:55 Atlanta 10.0.0.1 51594 Texas 10.0.0.25 80 HTTP UDP 4034 +2010-12-01 23:31:56 Atlanta 10.0.0.1 57734 Texas 10.0.0.26 80 HTTP UDP 3990 +2010-12-01 23:31:57 Atlanta 10.0.0.1 51717 Texas 10.0.0.25 80 HTTP UDP 2611 +2010-12-01 23:31:58 Atlanta 10.0.0.1 60987 Texas 10.0.0.27 80 HTTP UDP 2563 +2010-12-01 23:31:59 Atlanta 10.0.0.1 53737 Texas 10.0.0.28 80 HTTP UDP 5810 +2010-12-01 23:32:00 Atlanta 10.0.0.1 59365 Texas 10.0.0.30 80 HTTP UDP 2366 +2010-12-01 23:32:01 Atlanta 10.0.0.1 61255 Texas 10.0.0.25 80 HTTP UDP 4179 +2010-12-01 23:32:02 Atlanta 10.0.0.1 60042 Texas 10.0.0.30 80 HTTP UDP 5911 +2010-12-01 23:32:03 Atlanta 10.0.0.1 64649 Texas 10.0.0.29 80 HTTP UDP 4359 +2010-12-01 23:32:04 Atlanta 10.0.0.1 52784 Texas 10.0.0.28 80 HTTP UDP 4478 +2010-12-01 23:32:05 Atlanta 10.0.0.1 51000 Texas 10.0.0.29 80 HTTP UDP 4787 +2010-12-01 23:32:06 Atlanta 10.0.0.1 55785 Texas 10.0.0.28 80 HTTP UDP 2757 +2010-12-01 23:32:07 Atlanta 10.0.0.1 52545 Texas 10.0.0.29 80 HTTP UDP 4849 +2010-12-01 23:32:08 Atlanta 10.0.0.1 53041 Texas 10.0.0.25 80 HTTP UDP 4717 +2010-12-01 23:32:09 Atlanta 10.0.0.1 56784 Texas 10.0.0.30 80 HTTP UDP 4549 +2010-12-01 23:32:10 Atlanta 10.0.0.1 61366 Texas 10.0.0.27 80 HTTP UDP 4801 +2010-12-01 23:32:11 Atlanta 10.0.0.1 51827 Texas 10.0.0.26 80 HTTP UDP 2819 +2010-12-01 23:32:12 Atlanta 10.0.0.1 57950 Texas 10.0.0.28 80 HTTP UDP 2125 +2010-12-01 23:32:13 Atlanta 10.0.0.1 65439 Texas 10.0.0.27 80 HTTP UDP 4032 +2010-12-01 23:32:14 Atlanta 10.0.0.1 54184 Texas 10.0.0.28 80 HTTP UDP 3306 +2010-12-01 23:32:15 Atlanta 10.0.0.1 56505 Texas 10.0.0.25 80 HTTP UDP 5284 +2010-12-01 23:32:16 Atlanta 10.0.0.1 51770 Texas 10.0.0.25 80 HTTP UDP 4744 +2010-12-01 23:32:17 Atlanta 10.0.0.1 65072 Texas 10.0.0.26 80 HTTP UDP 4153 +2010-12-01 23:32:18 Atlanta 10.0.0.1 58809 Texas 10.0.0.29 80 HTTP UDP 3135 +2010-12-01 23:32:19 Atlanta 10.0.0.1 56167 Texas 10.0.0.29 80 HTTP UDP 5004 +2010-12-01 23:32:20 Atlanta 10.0.0.1 60502 Texas 10.0.0.28 80 HTTP UDP 4501 +2010-12-01 23:32:21 Atlanta 10.0.0.1 49402 Texas 10.0.0.28 80 HTTP UDP 2312 +2010-12-01 23:32:22 Atlanta 10.0.0.1 63766 Texas 10.0.0.29 80 HTTP UDP 3480 +2010-12-01 23:32:23 Atlanta 10.0.0.1 53118 Texas 10.0.0.29 80 HTTP UDP 3789 +2010-12-01 23:32:24 Atlanta 10.0.0.1 63214 Texas 10.0.0.29 80 HTTP UDP 5232 +2010-12-01 23:32:25 Atlanta 10.0.0.1 53701 Texas 10.0.0.30 80 HTTP UDP 3875 +2010-12-01 23:32:26 Atlanta 10.0.0.1 57085 Texas 10.0.0.26 80 HTTP UDP 4041 +2010-12-01 23:32:27 Atlanta 10.0.0.1 55509 Texas 10.0.0.28 80 HTTP UDP 4189 +2010-12-01 23:32:28 Atlanta 10.0.0.1 62648 Texas 10.0.0.29 80 HTTP UDP 4849 +2010-12-01 23:32:29 Atlanta 10.0.0.1 64419 Texas 10.0.0.25 80 HTTP UDP 5009 +2010-12-01 23:32:30 Atlanta 10.0.0.1 64494 Texas 10.0.0.30 80 HTTP UDP 2391 +2010-12-01 23:32:31 Atlanta 10.0.0.1 63422 Texas 10.0.0.27 80 HTTP UDP 4774 +2010-12-01 23:32:32 Atlanta 10.0.0.1 59952 Texas 10.0.0.28 80 HTTP UDP 4644 +2010-12-01 23:32:33 Atlanta 10.0.0.1 59513 Texas 10.0.0.27 80 HTTP UDP 5720 +2010-12-01 23:32:34 Atlanta 10.0.0.1 63763 Texas 10.0.0.25 80 HTTP UDP 2579 +2010-12-01 23:32:35 Atlanta 10.0.0.1 59039 Texas 10.0.0.27 80 HTTP UDP 3860 +2010-12-01 23:32:36 Atlanta 10.0.0.1 49564 Texas 10.0.0.25 80 HTTP UDP 4683 +2010-12-01 23:32:37 Atlanta 10.0.0.1 49845 Texas 10.0.0.28 80 HTTP UDP 5371 +2010-12-01 23:32:38 Atlanta 10.0.0.1 61084 Texas 10.0.0.30 80 HTTP UDP 2617 +2010-12-01 23:32:39 Atlanta 10.0.0.1 50344 Texas 10.0.0.28 80 HTTP UDP 5946 +2010-12-01 23:32:40 Atlanta 10.0.0.1 54168 Texas 10.0.0.30 80 HTTP UDP 5268 +2010-12-01 23:32:41 Atlanta 10.0.0.1 61089 Texas 10.0.0.30 80 HTTP UDP 5082 +2010-12-01 23:32:42 Atlanta 10.0.0.1 49692 Texas 10.0.0.26 80 HTTP UDP 2963 +2010-12-01 23:32:43 Atlanta 10.0.0.1 59905 Texas 10.0.0.30 80 HTTP UDP 2403 +2010-12-01 23:32:44 Atlanta 10.0.0.1 52396 Texas 10.0.0.27 80 HTTP UDP 3814 +2010-12-01 23:32:45 Atlanta 10.0.0.1 64420 Texas 10.0.0.29 80 HTTP UDP 2694 +2010-12-01 23:32:46 Atlanta 10.0.0.1 59262 Texas 10.0.0.25 80 HTTP UDP 5019 +2010-12-01 23:32:47 Atlanta 10.0.0.1 54399 Texas 10.0.0.29 80 HTTP UDP 5966 +2010-12-01 23:32:48 Atlanta 10.0.0.1 58522 Texas 10.0.0.26 80 HTTP UDP 5694 +2010-12-01 23:32:49 Atlanta 10.0.0.1 61656 Texas 10.0.0.28 80 HTTP UDP 2909 +2010-12-01 23:32:50 Atlanta 10.0.0.1 52950 Texas 10.0.0.28 80 HTTP UDP 4801 +2010-12-01 23:32:51 Atlanta 10.0.0.1 55165 Texas 10.0.0.29 80 HTTP UDP 2580 +2010-12-01 23:32:52 Atlanta 10.0.0.1 51559 Texas 10.0.0.26 80 HTTP UDP 3431 +2010-12-01 23:32:53 Atlanta 10.0.0.1 61385 Texas 10.0.0.27 80 HTTP UDP 3183 +2010-12-01 23:32:54 Atlanta 10.0.0.1 50947 Texas 10.0.0.25 80 HTTP UDP 5345 +2010-12-01 23:32:55 Atlanta 10.0.0.1 52069 Texas 10.0.0.25 80 HTTP UDP 3310 +2010-12-01 23:32:56 Atlanta 10.0.0.1 54968 Texas 10.0.0.25 80 HTTP UDP 2079 +2010-12-01 23:32:57 Atlanta 10.0.0.1 52058 Texas 10.0.0.25 80 HTTP UDP 2595 +2010-12-01 23:32:58 Atlanta 10.0.0.1 54273 Texas 10.0.0.25 80 HTTP UDP 2002 +2010-12-01 23:32:59 Atlanta 10.0.0.1 64509 Texas 10.0.0.28 80 HTTP UDP 5472 +2010-12-01 23:33:00 Atlanta 10.0.0.1 52815 Texas 10.0.0.29 80 HTTP UDP 3224 +2010-12-01 23:33:01 Atlanta 10.0.0.1 64927 Texas 10.0.0.27 80 HTTP UDP 2652 +2010-12-01 23:33:02 Atlanta 10.0.0.1 51621 Texas 10.0.0.30 80 HTTP UDP 4879 +2010-12-01 23:33:03 Atlanta 10.0.0.1 60476 Texas 10.0.0.25 80 HTTP UDP 3499 +2010-12-01 23:33:04 Atlanta 10.0.0.1 61844 Texas 10.0.0.28 80 HTTP UDP 3607 +2010-12-01 23:33:05 Atlanta 10.0.0.1 64912 Texas 10.0.0.27 80 HTTP UDP 4933 +2010-12-01 23:33:06 Atlanta 10.0.0.1 63194 Texas 10.0.0.28 80 HTTP UDP 5285 +2010-12-01 23:33:07 Atlanta 10.0.0.1 58688 Texas 10.0.0.28 80 HTTP UDP 5284 +2010-12-01 23:33:08 Atlanta 10.0.0.1 63872 Texas 10.0.0.30 80 HTTP UDP 5039 +2010-12-01 23:33:09 Atlanta 10.0.0.1 62136 Texas 10.0.0.29 80 HTTP UDP 3860 +2010-12-01 23:33:10 Atlanta 10.0.0.1 63993 Texas 10.0.0.27 80 HTTP UDP 3445 +2010-12-01 23:33:11 Atlanta 10.0.0.1 50168 Texas 10.0.0.26 80 HTTP UDP 5043 +2010-12-01 23:33:12 Atlanta 10.0.0.1 62968 Texas 10.0.0.26 80 HTTP UDP 5280 +2010-12-01 23:33:13 Atlanta 10.0.0.1 60042 Texas 10.0.0.26 80 HTTP UDP 4270 +2010-12-01 23:33:14 Atlanta 10.0.0.1 64970 Texas 10.0.0.30 80 HTTP UDP 4724 +2010-12-01 23:33:15 Atlanta 10.0.0.1 62410 Texas 10.0.0.29 80 HTTP UDP 4647 +2010-12-01 23:33:16 Atlanta 10.0.0.1 54683 Texas 10.0.0.28 80 HTTP UDP 4312 +2010-12-01 23:33:17 Atlanta 10.0.0.1 59617 Texas 10.0.0.29 80 HTTP UDP 4046 +2010-12-01 23:33:18 Atlanta 10.0.0.1 56728 Texas 10.0.0.29 80 HTTP UDP 4936 +2010-12-01 23:33:19 Atlanta 10.0.0.1 50842 Texas 10.0.0.29 80 HTTP UDP 4188 +2010-12-01 23:33:20 Atlanta 10.0.0.1 60583 Texas 10.0.0.28 80 HTTP UDP 5355 +2010-12-01 23:33:21 Atlanta 10.0.0.1 51039 Texas 10.0.0.25 80 HTTP UDP 3855 +2010-12-01 23:33:22 Atlanta 10.0.0.1 62656 Texas 10.0.0.25 80 HTTP UDP 5309 +2010-12-01 23:33:23 Atlanta 10.0.0.1 64339 Texas 10.0.0.28 80 HTTP UDP 4395 +2010-12-01 23:33:24 Atlanta 10.0.0.1 62540 Texas 10.0.0.28 80 HTTP UDP 3347 +2010-12-01 23:33:25 Atlanta 10.0.0.1 57415 Texas 10.0.0.29 80 HTTP UDP 3292 +2010-12-01 23:33:26 Atlanta 10.0.0.1 54213 Texas 10.0.0.29 80 HTTP UDP 5369 +2010-12-01 23:33:27 Atlanta 10.0.0.1 62435 Texas 10.0.0.27 80 HTTP UDP 5267 +2010-12-01 23:33:28 Atlanta 10.0.0.1 52944 Texas 10.0.0.28 80 HTTP UDP 3421 +2010-12-01 23:33:29 Atlanta 10.0.0.1 53120 Texas 10.0.0.27 80 HTTP UDP 4374 +2010-12-01 23:33:30 Atlanta 10.0.0.1 56652 Texas 10.0.0.25 80 HTTP UDP 2329 +2010-12-01 23:33:31 Atlanta 10.0.0.1 49940 Texas 10.0.0.25 80 HTTP UDP 4273 +2010-12-01 23:33:32 Atlanta 10.0.0.1 51020 Texas 10.0.0.25 80 HTTP UDP 4270 +2010-12-01 23:33:33 Atlanta 10.0.0.1 56163 Texas 10.0.0.25 80 HTTP UDP 2946 +2010-12-01 23:33:34 Atlanta 10.0.0.1 52870 Texas 10.0.0.27 80 HTTP UDP 2231 +2010-12-01 23:33:35 Atlanta 10.0.0.1 49428 Texas 10.0.0.30 80 HTTP UDP 3889 +2010-12-01 23:33:36 Atlanta 10.0.0.1 55495 Texas 10.0.0.26 80 HTTP UDP 2994 +2010-12-01 23:33:37 Atlanta 10.0.0.1 49954 Texas 10.0.0.29 80 HTTP UDP 3331 +2010-12-01 23:33:38 Atlanta 10.0.0.1 57299 Texas 10.0.0.26 80 HTTP UDP 2366 +2010-12-01 23:33:39 Atlanta 10.0.0.1 56875 Texas 10.0.0.26 80 HTTP UDP 4113 +2010-12-01 23:33:40 Atlanta 10.0.0.1 52453 Texas 10.0.0.28 80 HTTP UDP 2263 +2010-12-01 23:33:41 Atlanta 10.0.0.1 55425 Texas 10.0.0.30 80 HTTP UDP 5171 +2010-12-01 23:33:42 Atlanta 10.0.0.1 52636 Texas 10.0.0.26 80 HTTP UDP 3905 +2010-12-01 23:33:43 Atlanta 10.0.0.1 64166 Texas 10.0.0.28 80 HTTP UDP 4683 +2010-12-01 23:33:44 Atlanta 10.0.0.1 64415 Texas 10.0.0.30 80 HTTP UDP 5841 +2010-12-01 23:33:45 Atlanta 10.0.0.1 57475 Texas 10.0.0.25 80 HTTP UDP 2860 +2010-12-01 23:33:46 Atlanta 10.0.0.1 60695 Texas 10.0.0.29 80 HTTP UDP 2997 +2010-12-01 23:33:47 Atlanta 10.0.0.1 61162 Texas 10.0.0.30 80 HTTP UDP 2690 +2010-12-01 23:33:48 Atlanta 10.0.0.1 61822 Texas 10.0.0.26 80 HTTP UDP 2217 +2010-12-01 23:33:49 Atlanta 10.0.0.1 50166 Texas 10.0.0.30 80 HTTP UDP 4530 +2010-12-01 23:33:50 Atlanta 10.0.0.1 56666 Texas 10.0.0.27 80 HTTP UDP 3893 +2010-12-01 23:33:51 Atlanta 10.0.0.1 59113 Texas 10.0.0.30 80 HTTP UDP 2114 +2010-12-01 23:33:52 Atlanta 10.0.0.1 49449 Texas 10.0.0.25 80 HTTP UDP 4488 +2010-12-01 23:33:53 Atlanta 10.0.0.1 62386 Texas 10.0.0.25 80 HTTP UDP 4538 +2010-12-01 23:33:54 Atlanta 10.0.0.1 61508 Texas 10.0.0.28 80 HTTP UDP 4879 +2010-12-01 23:33:55 Atlanta 10.0.0.1 50007 Texas 10.0.0.25 80 HTTP UDP 2882 +2010-12-01 23:33:56 Atlanta 10.0.0.1 51996 Texas 10.0.0.29 80 HTTP UDP 2891 +2010-12-01 23:33:57 Atlanta 10.0.0.1 51036 Texas 10.0.0.27 80 HTTP UDP 5918 +2010-12-01 23:33:58 Atlanta 10.0.0.1 51348 Texas 10.0.0.27 80 HTTP UDP 5826 +2010-12-01 23:33:59 Atlanta 10.0.0.1 55180 Texas 10.0.0.26 80 HTTP UDP 3970 +2010-12-01 23:34:00 Atlanta 10.0.0.1 52142 Texas 10.0.0.25 80 HTTP UDP 2797 +2010-12-01 23:34:01 Atlanta 10.0.0.1 61518 Texas 10.0.0.25 80 HTTP UDP 2298 +2010-12-01 23:34:02 Atlanta 10.0.0.1 57145 Texas 10.0.0.30 80 HTTP UDP 3647 +2010-12-01 23:34:03 Atlanta 10.0.0.1 51030 Texas 10.0.0.28 80 HTTP UDP 2003 +2010-12-01 23:34:04 Atlanta 10.0.0.1 58506 Texas 10.0.0.27 80 HTTP UDP 5275 +2010-12-01 23:34:05 Atlanta 10.0.0.1 63738 Texas 10.0.0.29 80 HTTP UDP 2728 +2010-12-01 23:34:06 Atlanta 10.0.0.1 50635 Texas 10.0.0.25 80 HTTP UDP 5338 +2010-12-01 23:34:07 Atlanta 10.0.0.1 65046 Texas 10.0.0.30 80 HTTP UDP 4316 +2010-12-01 23:34:08 Atlanta 10.0.0.1 54373 Texas 10.0.0.30 80 HTTP UDP 3826 +2010-12-01 23:34:09 Atlanta 10.0.0.1 52070 Texas 10.0.0.29 80 HTTP UDP 4355 +2010-12-01 23:34:10 Atlanta 10.0.0.1 63148 Texas 10.0.0.25 80 HTTP UDP 3497 +2010-12-01 23:34:11 Atlanta 10.0.0.1 62331 Texas 10.0.0.27 80 HTTP UDP 3303 +2010-12-01 23:34:12 Atlanta 10.0.0.1 63851 Texas 10.0.0.26 80 HTTP UDP 2477 +2010-12-01 23:34:13 Atlanta 10.0.0.1 56793 Texas 10.0.0.26 80 HTTP UDP 2875 +2010-12-01 23:34:14 Atlanta 10.0.0.1 53425 Texas 10.0.0.29 80 HTTP UDP 2910 +2010-12-01 23:34:15 Atlanta 10.0.0.1 56012 Texas 10.0.0.30 80 HTTP UDP 2511 +2010-12-01 23:34:16 Atlanta 10.0.0.1 62344 Texas 10.0.0.29 80 HTTP UDP 4564 +2010-12-01 23:34:17 Atlanta 10.0.0.1 50014 Texas 10.0.0.29 80 HTTP UDP 3196 +2010-12-01 23:34:18 Atlanta 10.0.0.1 54082 Texas 10.0.0.29 80 HTTP UDP 5480 +2010-12-01 23:34:19 Atlanta 10.0.0.1 55718 Texas 10.0.0.25 80 HTTP UDP 2059 +2010-12-01 23:34:20 Atlanta 10.0.0.1 51271 Texas 10.0.0.30 80 HTTP UDP 5997 +2010-12-01 23:34:21 Atlanta 10.0.0.1 55871 Texas 10.0.0.28 80 HTTP UDP 2572 +2010-12-01 23:34:22 Atlanta 10.0.0.1 63577 Texas 10.0.0.27 80 HTTP UDP 3647 +2010-12-01 23:34:23 Atlanta 10.0.0.1 65163 Texas 10.0.0.30 80 HTTP UDP 5907 +2010-12-01 23:34:24 Atlanta 10.0.0.1 50765 Texas 10.0.0.27 80 HTTP UDP 3955 +2010-12-01 23:34:25 Atlanta 10.0.0.1 54968 Texas 10.0.0.25 80 HTTP UDP 3104 +2010-12-01 23:34:26 Atlanta 10.0.0.1 50590 Texas 10.0.0.27 80 HTTP UDP 2014 +2010-12-01 23:34:27 Atlanta 10.0.0.1 50801 Texas 10.0.0.28 80 HTTP UDP 4191 +2010-12-01 23:34:28 Atlanta 10.0.0.1 55565 Texas 10.0.0.26 80 HTTP UDP 5338 +2010-12-01 23:34:29 Atlanta 10.0.0.1 51862 Texas 10.0.0.28 80 HTTP UDP 4013 +2010-12-01 23:34:30 Atlanta 10.0.0.1 55351 Texas 10.0.0.29 80 HTTP UDP 2125 +2010-12-01 23:34:31 Atlanta 10.0.0.1 60771 Texas 10.0.0.29 80 HTTP UDP 4521 +2010-12-01 23:34:32 Atlanta 10.0.0.1 49867 Texas 10.0.0.29 80 HTTP UDP 5570 +2010-12-01 23:34:33 Atlanta 10.0.0.1 53762 Texas 10.0.0.29 80 HTTP UDP 2421 +2010-12-01 23:34:34 Atlanta 10.0.0.1 54810 Texas 10.0.0.29 80 HTTP UDP 3761 +2010-12-01 23:34:35 Atlanta 10.0.0.1 62783 Texas 10.0.0.27 80 HTTP UDP 3663 +2010-12-01 23:34:36 Atlanta 10.0.0.1 59009 Texas 10.0.0.28 80 HTTP UDP 2701 +2010-12-01 23:34:37 Atlanta 10.0.0.1 57534 Texas 10.0.0.25 80 HTTP UDP 4397 +2010-12-01 23:34:38 Atlanta 10.0.0.1 61777 Texas 10.0.0.28 80 HTTP UDP 3261 +2010-12-01 23:34:39 Atlanta 10.0.0.1 60671 Texas 10.0.0.29 80 HTTP UDP 5853 +2010-12-01 23:34:40 Atlanta 10.0.0.1 63430 Texas 10.0.0.30 80 HTTP UDP 2308 +2010-12-01 23:34:41 Atlanta 10.0.0.1 64696 Texas 10.0.0.27 80 HTTP UDP 2830 +2010-12-01 23:34:42 Atlanta 10.0.0.1 58168 Texas 10.0.0.29 80 HTTP UDP 3826 +2010-12-01 23:34:43 Atlanta 10.0.0.1 63276 Texas 10.0.0.30 80 HTTP UDP 2545 +2010-12-01 23:34:44 Atlanta 10.0.0.1 65444 Texas 10.0.0.25 80 HTTP UDP 5817 +2010-12-01 23:34:45 Atlanta 10.0.0.1 56180 Texas 10.0.0.25 80 HTTP UDP 2133 +2010-12-01 23:34:46 Atlanta 10.0.0.1 60364 Texas 10.0.0.26 80 HTTP UDP 4241 +2010-12-01 23:34:47 Atlanta 10.0.0.1 61290 Texas 10.0.0.25 80 HTTP UDP 4440 +2010-12-01 23:34:48 Atlanta 10.0.0.1 60208 Texas 10.0.0.25 80 HTTP UDP 3138 +2010-12-01 23:34:49 Atlanta 10.0.0.1 63645 Texas 10.0.0.27 80 HTTP UDP 2667 +2010-12-01 23:34:50 Atlanta 10.0.0.1 57805 Texas 10.0.0.29 80 HTTP UDP 4414 +2010-12-01 23:34:51 Atlanta 10.0.0.1 56140 Texas 10.0.0.29 80 HTTP UDP 4655 +2010-12-01 23:34:52 Atlanta 10.0.0.1 64438 Texas 10.0.0.30 80 HTTP UDP 3559 +2010-12-01 23:34:53 Atlanta 10.0.0.1 54065 Texas 10.0.0.29 80 HTTP UDP 5484 +2010-12-01 23:34:54 Atlanta 10.0.0.1 58370 Texas 10.0.0.28 80 HTTP UDP 5057 +2010-12-01 23:34:55 Atlanta 10.0.0.1 50505 Texas 10.0.0.26 80 HTTP UDP 3142 +2010-12-01 23:34:56 Atlanta 10.0.0.1 65525 Texas 10.0.0.25 80 HTTP UDP 2551 +2010-12-01 23:34:57 Atlanta 10.0.0.1 56006 Texas 10.0.0.25 80 HTTP UDP 2242 +2010-12-01 23:34:58 Atlanta 10.0.0.1 55287 Texas 10.0.0.27 80 HTTP UDP 2954 +2010-12-01 23:34:59 Atlanta 10.0.0.1 50242 Texas 10.0.0.26 80 HTTP UDP 2627 +2010-12-01 23:35:00 Atlanta 10.0.0.1 50579 Texas 10.0.0.27 80 HTTP UDP 5634 +2010-12-01 23:35:01 Atlanta 10.0.0.1 61300 Texas 10.0.0.27 80 HTTP UDP 3171 +2010-12-01 23:35:02 Atlanta 10.0.0.1 55219 Texas 10.0.0.28 80 HTTP UDP 5856 +2010-12-01 23:35:03 Atlanta 10.0.0.1 53239 Texas 10.0.0.27 80 HTTP UDP 5930 +2010-12-01 23:35:04 Atlanta 10.0.0.1 51734 Texas 10.0.0.27 80 HTTP UDP 5340 +2010-12-01 23:35:05 Atlanta 10.0.0.1 53966 Texas 10.0.0.28 80 HTTP UDP 2782 +2010-12-01 23:35:06 Atlanta 10.0.0.1 59575 Texas 10.0.0.27 80 HTTP UDP 4281 +2010-12-01 23:35:07 Atlanta 10.0.0.1 62593 Texas 10.0.0.29 80 HTTP UDP 4567 +2010-12-01 23:35:08 Atlanta 10.0.0.1 58679 Texas 10.0.0.28 80 HTTP UDP 5841 +2010-12-01 23:35:09 Atlanta 10.0.0.1 57896 Texas 10.0.0.26 80 HTTP UDP 3630 +2010-12-01 23:35:10 Atlanta 10.0.0.1 50359 Texas 10.0.0.28 80 HTTP UDP 3096 +2010-12-01 23:35:11 Atlanta 10.0.0.1 64939 Texas 10.0.0.25 80 HTTP UDP 4368 +2010-12-01 23:35:12 Atlanta 10.0.0.1 62013 Texas 10.0.0.27 80 HTTP UDP 3708 +2010-12-01 23:35:13 Atlanta 10.0.0.1 56764 Texas 10.0.0.29 80 HTTP UDP 3685 +2010-12-01 23:35:14 Atlanta 10.0.0.1 55790 Texas 10.0.0.29 80 HTTP UDP 4977 +2010-12-01 23:35:15 Atlanta 10.0.0.1 58672 Texas 10.0.0.29 80 HTTP UDP 5072 +2010-12-01 23:35:16 Atlanta 10.0.0.1 63273 Texas 10.0.0.27 80 HTTP UDP 3223 +2010-12-01 23:35:17 Atlanta 10.0.0.1 55107 Texas 10.0.0.29 80 HTTP UDP 4088 +2010-12-01 23:35:18 Atlanta 10.0.0.1 62888 Texas 10.0.0.25 80 HTTP UDP 4893 +2010-12-01 23:35:19 Atlanta 10.0.0.1 50355 Texas 10.0.0.30 80 HTTP UDP 5807 +2010-12-01 23:35:20 Atlanta 10.0.0.1 64087 Texas 10.0.0.30 80 HTTP UDP 3042 +2010-12-01 23:35:21 Atlanta 10.0.0.1 52567 Texas 10.0.0.29 80 HTTP UDP 2563 +2010-12-01 23:35:22 Atlanta 10.0.0.1 59169 Texas 10.0.0.30 80 HTTP UDP 4902 +2010-12-01 23:35:23 Atlanta 10.0.0.1 54878 Texas 10.0.0.28 80 HTTP UDP 3523 +2010-12-01 23:35:24 Atlanta 10.0.0.1 54842 Texas 10.0.0.27 80 HTTP UDP 5558 +2010-12-01 23:35:25 Atlanta 10.0.0.1 57320 Texas 10.0.0.26 80 HTTP UDP 5989 +2010-12-01 23:35:26 Atlanta 10.0.0.1 54756 Texas 10.0.0.28 80 HTTP UDP 4711 +2010-12-01 23:35:27 Atlanta 10.0.0.1 52185 Texas 10.0.0.27 80 HTTP UDP 2255 +2010-12-01 23:35:28 Atlanta 10.0.0.1 54745 Texas 10.0.0.27 80 HTTP UDP 4787 +2010-12-01 23:35:29 Atlanta 10.0.0.1 51617 Texas 10.0.0.29 80 HTTP UDP 5899 +2010-12-01 23:35:30 Atlanta 10.0.0.1 52261 Texas 10.0.0.26 80 HTTP UDP 3198 +2010-12-01 23:35:31 Atlanta 10.0.0.1 53618 Texas 10.0.0.25 80 HTTP UDP 2380 +2010-12-01 23:35:32 Atlanta 10.0.0.1 51225 Texas 10.0.0.28 80 HTTP UDP 5696 +2010-12-01 23:35:33 Atlanta 10.0.0.1 52701 Texas 10.0.0.26 80 HTTP UDP 4651 +2010-12-01 23:35:34 Atlanta 10.0.0.1 57726 Texas 10.0.0.26 80 HTTP UDP 3480 +2010-12-01 23:35:35 Atlanta 10.0.0.1 60786 Texas 10.0.0.26 80 HTTP UDP 4411 +2010-12-01 23:35:36 Atlanta 10.0.0.1 63910 Texas 10.0.0.27 80 HTTP UDP 2240 +2010-12-01 23:35:37 Atlanta 10.0.0.1 62349 Texas 10.0.0.28 80 HTTP UDP 4449 +2010-12-01 23:35:38 Atlanta 10.0.0.1 65338 Texas 10.0.0.29 80 HTTP UDP 3908 +2010-12-01 23:35:39 Atlanta 10.0.0.1 64083 Texas 10.0.0.25 80 HTTP UDP 3990 +2010-12-01 23:35:40 Atlanta 10.0.0.1 55415 Texas 10.0.0.28 80 HTTP UDP 4102 +2010-12-01 23:35:41 Atlanta 10.0.0.1 63531 Texas 10.0.0.26 80 HTTP UDP 5047 +2010-12-01 23:35:42 Atlanta 10.0.0.1 52756 Texas 10.0.0.27 80 HTTP UDP 4449 +2010-12-01 23:35:43 Atlanta 10.0.0.1 54205 Texas 10.0.0.27 80 HTTP UDP 3499 +2010-12-01 23:35:44 Atlanta 10.0.0.1 59392 Texas 10.0.0.27 80 HTTP UDP 4724 +2010-12-01 23:35:45 Atlanta 10.0.0.1 65071 Texas 10.0.0.29 80 HTTP UDP 5422 +2010-12-01 23:35:46 Atlanta 10.0.0.1 56063 Texas 10.0.0.25 80 HTTP UDP 3834 +2010-12-01 23:35:47 Atlanta 10.0.0.1 64622 Texas 10.0.0.27 80 HTTP UDP 4903 +2010-12-01 23:35:48 Atlanta 10.0.0.1 54054 Texas 10.0.0.25 80 HTTP UDP 3549 +2010-12-01 23:35:49 Atlanta 10.0.0.1 50371 Texas 10.0.0.29 80 HTTP UDP 2830 +2010-12-01 23:35:50 Atlanta 10.0.0.1 50055 Texas 10.0.0.27 80 HTTP UDP 3814 +2010-12-01 23:35:51 Atlanta 10.0.0.1 50830 Texas 10.0.0.28 80 HTTP UDP 5963 +2010-12-01 23:35:52 Atlanta 10.0.0.1 59822 Texas 10.0.0.28 80 HTTP UDP 3125 +2010-12-01 23:35:53 Atlanta 10.0.0.1 57620 Texas 10.0.0.28 80 HTTP UDP 5559 +2010-12-01 23:35:54 Atlanta 10.0.0.1 64368 Texas 10.0.0.25 80 HTTP UDP 2099 +2010-12-01 23:35:55 Atlanta 10.0.0.1 53892 Texas 10.0.0.26 80 HTTP UDP 2924 +2010-12-01 23:35:56 Atlanta 10.0.0.1 51961 Texas 10.0.0.29 80 HTTP UDP 5835 +2010-12-01 23:35:57 Atlanta 10.0.0.1 55886 Texas 10.0.0.27 80 HTTP UDP 4359 +2010-12-01 23:35:58 Atlanta 10.0.0.1 65323 Texas 10.0.0.30 80 HTTP UDP 2098 +2010-12-01 23:35:59 Atlanta 10.0.0.1 65192 Texas 10.0.0.25 80 HTTP UDP 3099 +2010-12-01 23:36:00 Atlanta 10.0.0.1 62817 Texas 10.0.0.30 80 HTTP UDP 2024 +2010-12-01 23:36:01 Atlanta 10.0.0.1 55214 Texas 10.0.0.25 80 HTTP UDP 2312 +2010-12-01 23:36:02 Atlanta 10.0.0.1 52651 Texas 10.0.0.30 80 HTTP UDP 2693 +2010-12-01 23:36:03 Atlanta 10.0.0.1 60851 Texas 10.0.0.25 80 HTTP UDP 4034 +2010-12-01 23:36:04 Atlanta 10.0.0.1 51275 Texas 10.0.0.25 80 HTTP UDP 4287 +2010-12-01 23:36:05 Atlanta 10.0.0.1 57951 Texas 10.0.0.27 80 HTTP UDP 4936 +2010-12-01 23:36:06 Atlanta 10.0.0.1 64987 Texas 10.0.0.30 80 HTTP UDP 4176 +2010-12-01 23:36:07 Atlanta 10.0.0.1 51375 Texas 10.0.0.26 80 HTTP UDP 5817 +2010-12-01 23:36:08 Atlanta 10.0.0.1 64631 Texas 10.0.0.26 80 HTTP UDP 4920 +2010-12-01 23:36:09 Atlanta 10.0.0.1 57788 Texas 10.0.0.27 80 HTTP UDP 3905 +2010-12-01 23:36:10 Atlanta 10.0.0.1 63788 Texas 10.0.0.29 80 HTTP UDP 4102 +2010-12-01 23:36:11 Atlanta 10.0.0.1 65312 Texas 10.0.0.29 80 HTTP UDP 4241 +2010-12-01 23:36:12 Atlanta 10.0.0.1 52625 Texas 10.0.0.26 80 HTTP UDP 5787 +2010-12-01 23:36:13 Atlanta 10.0.0.1 57516 Texas 10.0.0.29 80 HTTP UDP 5092 +2010-12-01 23:36:14 Atlanta 10.0.0.1 59906 Texas 10.0.0.26 80 HTTP UDP 5907 +2010-12-01 23:36:15 Atlanta 10.0.0.1 62494 Texas 10.0.0.28 80 HTTP UDP 3142 +2010-12-01 23:36:16 Atlanta 10.0.0.1 61119 Texas 10.0.0.29 80 HTTP UDP 3736 +2010-12-01 23:36:17 Atlanta 10.0.0.1 52122 Texas 10.0.0.25 80 HTTP UDP 3865 +2010-12-01 23:36:18 Atlanta 10.0.0.1 52138 Texas 10.0.0.27 80 HTTP UDP 5082 +2010-12-01 23:36:19 Atlanta 10.0.0.1 60379 Texas 10.0.0.26 80 HTTP UDP 3256 +2010-12-01 23:36:20 Atlanta 10.0.0.1 63295 Texas 10.0.0.27 80 HTTP UDP 3562 +2010-12-01 23:36:21 Atlanta 10.0.0.1 53088 Texas 10.0.0.28 80 HTTP UDP 3940 +2010-12-01 23:36:22 Atlanta 10.0.0.1 64567 Texas 10.0.0.30 80 HTTP UDP 2603 +2010-12-01 23:36:23 Atlanta 10.0.0.1 53829 Texas 10.0.0.28 80 HTTP UDP 2189 +2010-12-01 23:36:24 Atlanta 10.0.0.1 57471 Texas 10.0.0.26 80 HTTP UDP 3607 +2010-12-01 23:36:25 Atlanta 10.0.0.1 49671 Texas 10.0.0.30 80 HTTP UDP 4684 +2010-12-01 23:36:26 Atlanta 10.0.0.1 53674 Texas 10.0.0.30 80 HTTP UDP 2924 +2010-12-01 23:36:27 Atlanta 10.0.0.1 49397 Texas 10.0.0.28 80 HTTP UDP 5946 +2010-12-01 23:36:28 Atlanta 10.0.0.1 51542 Texas 10.0.0.30 80 HTTP UDP 3310 +2010-12-01 23:36:29 Atlanta 10.0.0.1 63508 Texas 10.0.0.29 80 HTTP UDP 4365 +2010-12-01 23:36:30 Atlanta 10.0.0.1 57511 Texas 10.0.0.30 80 HTTP UDP 3445 +2010-12-01 23:36:31 Atlanta 10.0.0.1 59105 Texas 10.0.0.29 80 HTTP UDP 3402 +2010-12-01 23:36:32 Atlanta 10.0.0.1 60210 Texas 10.0.0.26 80 HTTP UDP 5162 +2010-12-01 23:36:33 Atlanta 10.0.0.1 65014 Texas 10.0.0.28 80 HTTP UDP 4273 +2010-12-01 23:36:34 Atlanta 10.0.0.1 52398 Texas 10.0.0.28 80 HTTP UDP 4951 +2010-12-01 23:36:35 Atlanta 10.0.0.1 59547 Texas 10.0.0.27 80 HTTP UDP 2643 +2010-12-01 23:36:36 Atlanta 10.0.0.1 59505 Texas 10.0.0.25 80 HTTP UDP 4977 +2010-12-01 23:36:37 Atlanta 10.0.0.1 53412 Texas 10.0.0.25 80 HTTP UDP 2511 +2010-12-01 23:36:38 Atlanta 10.0.0.1 64289 Texas 10.0.0.25 80 HTTP UDP 5603 +2010-12-01 23:36:39 Atlanta 10.0.0.1 61735 Texas 10.0.0.28 80 HTTP UDP 3063 +2010-12-01 23:36:40 Atlanta 10.0.0.1 63149 Texas 10.0.0.30 80 HTTP UDP 3742 +2010-12-01 23:36:41 Atlanta 10.0.0.1 55168 Texas 10.0.0.29 80 HTTP UDP 2525 +2010-12-01 23:36:42 Atlanta 10.0.0.1 54376 Texas 10.0.0.26 80 HTTP UDP 4417 +2010-12-01 23:36:43 Atlanta 10.0.0.1 62298 Texas 10.0.0.30 80 HTTP UDP 5952 +2010-12-01 23:36:44 Atlanta 10.0.0.1 50337 Texas 10.0.0.28 80 HTTP UDP 2939 +2010-12-01 23:36:45 Atlanta 10.0.0.1 49860 Texas 10.0.0.26 80 HTTP UDP 4524 +2010-12-01 23:36:46 Atlanta 10.0.0.1 62535 Texas 10.0.0.28 80 HTTP UDP 5256 +2010-12-01 23:36:47 Atlanta 10.0.0.1 60965 Texas 10.0.0.26 80 HTTP UDP 5775 +2010-12-01 23:36:48 Atlanta 10.0.0.1 60986 Texas 10.0.0.28 80 HTTP UDP 3900 +2010-12-01 23:36:49 Atlanta 10.0.0.1 65091 Texas 10.0.0.26 80 HTTP UDP 2235 +2010-12-01 23:36:50 Atlanta 10.0.0.1 52435 Texas 10.0.0.28 80 HTTP UDP 2577 +2010-12-01 23:36:51 Atlanta 10.0.0.1 61774 Texas 10.0.0.29 80 HTTP UDP 3216 +2010-12-01 23:36:52 Atlanta 10.0.0.1 55096 Texas 10.0.0.28 80 HTTP UDP 4792 +2010-12-01 23:36:53 Atlanta 10.0.0.1 63000 Texas 10.0.0.27 80 HTTP UDP 2569 +2010-12-01 23:36:54 Atlanta 10.0.0.1 61979 Texas 10.0.0.29 80 HTTP UDP 4103 +2010-12-01 23:36:55 Atlanta 10.0.0.1 56850 Texas 10.0.0.28 80 HTTP UDP 4368 +2010-12-01 23:36:56 Atlanta 10.0.0.1 49245 Texas 10.0.0.25 80 HTTP UDP 3138 +2010-12-01 23:36:57 Atlanta 10.0.0.1 62011 Texas 10.0.0.25 80 HTTP UDP 2421 +2010-12-01 23:36:58 Atlanta 10.0.0.1 63775 Texas 10.0.0.27 80 HTTP UDP 2850 +2010-12-01 23:36:59 Atlanta 10.0.0.1 62523 Texas 10.0.0.30 80 HTTP UDP 5759 +2010-12-01 23:37:00 Atlanta 10.0.0.1 53385 Texas 10.0.0.25 80 HTTP UDP 2007 +2010-12-01 23:37:01 Atlanta 10.0.0.1 65478 Texas 10.0.0.30 80 HTTP UDP 4977 +2010-12-01 23:37:02 Atlanta 10.0.0.1 54040 Texas 10.0.0.25 80 HTTP UDP 2690 +2010-12-01 23:37:03 Atlanta 10.0.0.1 59949 Texas 10.0.0.30 80 HTTP UDP 4801 +2010-12-01 23:37:04 Atlanta 10.0.0.1 49155 Texas 10.0.0.27 80 HTTP UDP 3267 +2010-12-01 23:37:05 Atlanta 10.0.0.1 63213 Texas 10.0.0.28 80 HTTP UDP 5727 +2010-12-01 23:37:06 Atlanta 10.0.0.1 64045 Texas 10.0.0.29 80 HTTP UDP 4774 +2010-12-01 23:37:07 Atlanta 10.0.0.1 62037 Texas 10.0.0.26 80 HTTP UDP 3934 +2010-12-01 23:37:08 Atlanta 10.0.0.1 60228 Texas 10.0.0.30 80 HTTP UDP 5938 +2010-12-01 23:37:09 Atlanta 10.0.0.1 60809 Texas 10.0.0.29 80 HTTP UDP 4042 +2010-12-01 23:37:10 Atlanta 10.0.0.1 55088 Texas 10.0.0.26 80 HTTP UDP 4192 +2010-12-01 23:37:11 Atlanta 10.0.0.1 64759 Texas 10.0.0.26 80 HTTP UDP 3302 +2010-12-01 23:37:12 Atlanta 10.0.0.1 57937 Texas 10.0.0.27 80 HTTP UDP 2652 +2010-12-01 23:37:13 Atlanta 10.0.0.1 55969 Texas 10.0.0.25 80 HTTP UDP 3618 +2010-12-01 23:37:14 Atlanta 10.0.0.1 62264 Texas 10.0.0.25 80 HTTP UDP 4969 +2010-12-01 23:37:15 Atlanta 10.0.0.1 54480 Texas 10.0.0.25 80 HTTP UDP 3970 +2010-12-01 23:37:16 Atlanta 10.0.0.1 61220 Texas 10.0.0.26 80 HTTP UDP 4238 +2010-12-01 23:37:17 Atlanta 10.0.0.1 54501 Texas 10.0.0.27 80 HTTP UDP 2997 +2010-12-01 23:37:18 Atlanta 10.0.0.1 63265 Texas 10.0.0.30 80 HTTP UDP 5997 +2010-12-01 23:37:19 Atlanta 10.0.0.1 65029 Texas 10.0.0.30 80 HTTP UDP 2312 +2010-12-01 23:37:20 Atlanta 10.0.0.1 57794 Texas 10.0.0.27 80 HTTP UDP 2782 +2010-12-01 23:37:21 Atlanta 10.0.0.1 57977 Texas 10.0.0.25 80 HTTP UDP 2757 +2010-12-01 23:37:22 Atlanta 10.0.0.1 63079 Texas 10.0.0.27 80 HTTP UDP 3198 +2010-12-01 23:37:23 Atlanta 10.0.0.1 52327 Texas 10.0.0.29 80 HTTP UDP 3562 +2010-12-01 23:37:24 Atlanta 10.0.0.1 54450 Texas 10.0.0.26 80 HTTP UDP 5002 +2010-12-01 23:37:25 Atlanta 10.0.0.1 51270 Texas 10.0.0.25 80 HTTP UDP 5950 +2010-12-01 23:37:26 Atlanta 10.0.0.1 63783 Texas 10.0.0.29 80 HTTP UDP 4461 +2010-12-01 23:37:27 Atlanta 10.0.0.1 60483 Texas 10.0.0.28 80 HTTP UDP 3331 +2010-12-01 23:37:28 Atlanta 10.0.0.1 58977 Texas 10.0.0.28 80 HTTP UDP 2997 +2010-12-01 23:37:29 Atlanta 10.0.0.1 65429 Texas 10.0.0.29 80 HTTP UDP 4717 +2010-12-01 23:37:30 Atlanta 10.0.0.1 50767 Texas 10.0.0.30 80 HTTP UDP 3538 +2010-12-01 23:37:31 Atlanta 10.0.0.1 57202 Texas 10.0.0.30 80 HTTP UDP 2640 +2010-12-01 23:37:32 Atlanta 10.0.0.1 60006 Texas 10.0.0.26 80 HTTP UDP 5284 +2010-12-01 23:37:33 Atlanta 10.0.0.1 60498 Texas 10.0.0.28 80 HTTP UDP 5774 +2010-12-01 23:37:34 Atlanta 10.0.0.1 51017 Texas 10.0.0.25 80 HTTP UDP 3070 +2010-12-01 23:37:35 Atlanta 10.0.0.1 58212 Texas 10.0.0.30 80 HTTP UDP 3211 +2010-12-01 23:37:36 Atlanta 10.0.0.1 51880 Texas 10.0.0.29 80 HTTP UDP 2432 +2010-12-01 23:37:37 Atlanta 10.0.0.1 61872 Texas 10.0.0.25 80 HTTP UDP 4064 +2010-12-01 23:37:38 Atlanta 10.0.0.1 53496 Texas 10.0.0.28 80 HTTP UDP 4867 +2010-12-01 23:37:39 Atlanta 10.0.0.1 60399 Texas 10.0.0.26 80 HTTP UDP 5022 +2010-12-01 23:37:40 Atlanta 10.0.0.1 54016 Texas 10.0.0.30 80 HTTP UDP 4138 +2010-12-01 23:37:41 Atlanta 10.0.0.1 64059 Texas 10.0.0.26 80 HTTP UDP 4638 +2010-12-01 23:37:42 Atlanta 10.0.0.1 56463 Texas 10.0.0.27 80 HTTP UDP 4903 +2010-12-01 23:37:43 Atlanta 10.0.0.1 54669 Texas 10.0.0.26 80 HTTP UDP 2946 +2010-12-01 23:37:44 Atlanta 10.0.0.1 58761 Texas 10.0.0.28 80 HTTP UDP 2272 +2010-12-01 23:37:45 Atlanta 10.0.0.1 61818 Texas 10.0.0.27 80 HTTP UDP 3834 +2010-12-01 23:37:46 Atlanta 10.0.0.1 53272 Texas 10.0.0.27 80 HTTP UDP 4935 +2010-12-01 23:37:47 Atlanta 10.0.0.1 57425 Texas 10.0.0.27 80 HTTP UDP 2931 +2010-12-01 23:37:48 Atlanta 10.0.0.1 65047 Texas 10.0.0.30 80 HTTP UDP 2836 +2010-12-01 23:37:49 Atlanta 10.0.0.1 53048 Texas 10.0.0.29 80 HTTP UDP 4253 +2010-12-01 23:37:50 Atlanta 10.0.0.1 63325 Texas 10.0.0.30 80 HTTP UDP 4041 +2010-12-01 23:37:51 Atlanta 10.0.0.1 65112 Texas 10.0.0.29 80 HTTP UDP 5252 +2010-12-01 23:37:52 Atlanta 10.0.0.1 49448 Texas 10.0.0.28 80 HTTP UDP 4103 +2010-12-01 23:37:53 Atlanta 10.0.0.1 58545 Texas 10.0.0.28 80 HTTP UDP 5888 +2010-12-01 23:37:54 Atlanta 10.0.0.1 64048 Texas 10.0.0.26 80 HTTP UDP 4955 +2010-12-01 23:37:55 Atlanta 10.0.0.1 56425 Texas 10.0.0.27 80 HTTP UDP 4359 +2010-12-01 23:37:56 Atlanta 10.0.0.1 50247 Texas 10.0.0.26 80 HTTP UDP 2844 +2010-12-01 23:37:57 Atlanta 10.0.0.1 52044 Texas 10.0.0.28 80 HTTP UDP 5472 +2010-12-01 23:37:58 Atlanta 10.0.0.1 54693 Texas 10.0.0.29 80 HTTP UDP 5694 +2010-12-01 23:37:59 Atlanta 10.0.0.1 60979 Texas 10.0.0.29 80 HTTP UDP 2690 +2010-12-01 23:38:00 Atlanta 10.0.0.1 54628 Texas 10.0.0.26 80 HTTP UDP 4817 +2010-12-01 23:38:01 Atlanta 10.0.0.1 60578 Texas 10.0.0.26 80 HTTP UDP 2860 +2010-12-01 23:38:02 Atlanta 10.0.0.1 57443 Texas 10.0.0.27 80 HTTP UDP 4032 +2010-12-01 23:38:03 Atlanta 10.0.0.1 58081 Texas 10.0.0.26 80 HTTP UDP 4281 +2010-12-01 23:38:04 Atlanta 10.0.0.1 50081 Texas 10.0.0.29 80 HTTP UDP 3934 +2010-12-01 23:38:05 Atlanta 10.0.0.1 63765 Texas 10.0.0.26 80 HTTP UDP 5673 +2010-12-01 23:38:06 Atlanta 10.0.0.1 52617 Texas 10.0.0.29 80 HTTP UDP 4414 +2010-12-01 23:38:07 Atlanta 10.0.0.1 55113 Texas 10.0.0.28 80 HTTP UDP 2294 +2010-12-01 23:38:08 Atlanta 10.0.0.1 64176 Texas 10.0.0.25 80 HTTP UDP 4777 +2010-12-01 23:38:09 Atlanta 10.0.0.1 63848 Texas 10.0.0.26 80 HTTP UDP 5888 +2010-12-01 23:38:10 Atlanta 10.0.0.1 54058 Texas 10.0.0.28 80 HTTP UDP 5841 +2010-12-01 23:38:11 Atlanta 10.0.0.1 49762 Texas 10.0.0.29 80 HTTP UDP 3959 +2010-12-01 23:38:12 Atlanta 10.0.0.1 57310 Texas 10.0.0.26 80 HTTP UDP 3806 +2010-12-01 23:38:13 Atlanta 10.0.0.1 64210 Texas 10.0.0.28 80 HTTP UDP 5015 +2010-12-01 23:38:14 Atlanta 10.0.0.1 58743 Texas 10.0.0.30 80 HTTP UDP 3357 +2010-12-01 23:38:15 Atlanta 10.0.0.1 54300 Texas 10.0.0.29 80 HTTP UDP 2690 +2010-12-01 23:38:16 Atlanta 10.0.0.1 52996 Texas 10.0.0.28 80 HTTP UDP 3761 +2010-12-01 23:38:17 Atlanta 10.0.0.1 53011 Texas 10.0.0.28 80 HTTP UDP 5841 +2010-12-01 23:38:18 Atlanta 10.0.0.1 50972 Texas 10.0.0.30 80 HTTP UDP 4251 +2010-12-01 23:38:19 Atlanta 10.0.0.1 57003 Texas 10.0.0.27 80 HTTP UDP 5284 +2010-12-01 23:38:20 Atlanta 10.0.0.1 53785 Texas 10.0.0.29 80 HTTP UDP 3900 +2010-12-01 23:38:21 Atlanta 10.0.0.1 55104 Texas 10.0.0.26 80 HTTP UDP 2994 +2010-12-01 23:38:22 Atlanta 10.0.0.1 63371 Texas 10.0.0.27 80 HTTP UDP 3421 +2010-12-01 23:38:23 Atlanta 10.0.0.1 61023 Texas 10.0.0.28 80 HTTP UDP 4293 +2010-12-01 23:38:24 Atlanta 10.0.0.1 62778 Texas 10.0.0.28 80 HTTP UDP 4717 +2010-12-01 23:38:25 Atlanta 10.0.0.1 55336 Texas 10.0.0.25 80 HTTP UDP 4653 +2010-12-01 23:38:26 Atlanta 10.0.0.1 54898 Texas 10.0.0.26 80 HTTP UDP 2231 +2010-12-01 23:38:27 Atlanta 10.0.0.1 55570 Texas 10.0.0.29 80 HTTP UDP 5849 +2010-12-01 23:38:28 Atlanta 10.0.0.1 49782 Texas 10.0.0.25 80 HTTP UDP 3559 +2010-12-01 23:38:29 Atlanta 10.0.0.1 56731 Texas 10.0.0.28 80 HTTP UDP 5096 +2010-12-01 23:38:30 Atlanta 10.0.0.1 51080 Texas 10.0.0.30 80 HTTP UDP 5775 +2010-12-01 23:38:31 Atlanta 10.0.0.1 58375 Texas 10.0.0.28 80 HTTP UDP 4461 +2010-12-01 23:38:32 Atlanta 10.0.0.1 56765 Texas 10.0.0.26 80 HTTP UDP 2422 +2010-12-01 23:38:33 Atlanta 10.0.0.1 58094 Texas 10.0.0.30 80 HTTP UDP 2972 +2010-12-01 23:38:34 Atlanta 10.0.0.1 51410 Texas 10.0.0.25 80 HTTP UDP 3540 +2010-12-01 23:38:35 Atlanta 10.0.0.1 52210 Texas 10.0.0.29 80 HTTP UDP 4657 +2010-12-01 23:38:36 Atlanta 10.0.0.1 52399 Texas 10.0.0.30 80 HTTP UDP 3585 +2010-12-01 23:38:37 Atlanta 10.0.0.1 55483 Texas 10.0.0.25 80 HTTP UDP 4728 +2010-12-01 23:38:38 Atlanta 10.0.0.1 64423 Texas 10.0.0.28 80 HTTP UDP 5837 +2010-12-01 23:38:39 Atlanta 10.0.0.1 63463 Texas 10.0.0.25 80 HTTP UDP 5542 +2010-12-01 23:38:40 Atlanta 10.0.0.1 57094 Texas 10.0.0.29 80 HTTP UDP 2312 +2010-12-01 23:38:41 Atlanta 10.0.0.1 56965 Texas 10.0.0.28 80 HTTP UDP 5119 +2010-12-01 23:38:42 Atlanta 10.0.0.1 59310 Texas 10.0.0.30 80 HTTP UDP 2836 +2010-12-01 23:38:43 Atlanta 10.0.0.1 58022 Texas 10.0.0.28 80 HTTP UDP 2813 +2010-12-01 23:38:44 Atlanta 10.0.0.1 60073 Texas 10.0.0.25 80 HTTP UDP 3997 +2010-12-01 23:38:45 Atlanta 10.0.0.1 51551 Texas 10.0.0.27 80 HTTP UDP 2391 +2010-12-01 23:38:46 Atlanta 10.0.0.1 55383 Texas 10.0.0.29 80 HTTP UDP 2652 +2010-12-01 23:38:47 Atlanta 10.0.0.1 56732 Texas 10.0.0.29 80 HTTP UDP 2857 +2010-12-01 23:38:48 Atlanta 10.0.0.1 61147 Texas 10.0.0.29 80 HTTP UDP 5594 +2010-12-01 23:38:49 Atlanta 10.0.0.1 54365 Texas 10.0.0.29 80 HTTP UDP 5029 +2010-12-01 23:38:50 Atlanta 10.0.0.1 57299 Texas 10.0.0.28 80 HTTP UDP 2860 +2010-12-01 23:38:51 Atlanta 10.0.0.1 59514 Texas 10.0.0.26 80 HTTP UDP 2226 +2010-12-01 23:38:52 Atlanta 10.0.0.1 65428 Texas 10.0.0.26 80 HTTP UDP 2221 +2010-12-01 23:38:53 Atlanta 10.0.0.1 61884 Texas 10.0.0.28 80 HTTP UDP 5749 +2010-12-01 23:38:54 Atlanta 10.0.0.1 58075 Texas 10.0.0.27 80 HTTP UDP 4179 +2010-12-01 23:38:55 Atlanta 10.0.0.1 61385 Texas 10.0.0.28 80 HTTP UDP 2855 +2010-12-01 23:38:56 Atlanta 10.0.0.1 65087 Texas 10.0.0.30 80 HTTP UDP 5472 +2010-12-01 23:38:57 Atlanta 10.0.0.1 52336 Texas 10.0.0.30 80 HTTP UDP 5917 +2010-12-01 23:38:58 Atlanta 10.0.0.1 62411 Texas 10.0.0.26 80 HTTP UDP 5738 +2010-12-01 23:38:59 Atlanta 10.0.0.1 57329 Texas 10.0.0.26 80 HTTP UDP 4683 +2010-12-01 23:39:00 Atlanta 10.0.0.1 55423 Texas 10.0.0.28 80 HTTP UDP 5096 +2010-12-01 23:39:01 Atlanta 10.0.0.1 57431 Texas 10.0.0.30 80 HTTP UDP 3326 +2010-12-01 23:39:02 Atlanta 10.0.0.1 55723 Texas 10.0.0.30 80 HTTP UDP 5258 +2010-12-01 23:39:03 Atlanta 10.0.0.1 58364 Texas 10.0.0.26 80 HTTP UDP 3421 +2010-12-01 23:39:04 Atlanta 10.0.0.1 55307 Texas 10.0.0.26 80 HTTP UDP 5412 +2010-12-01 23:39:05 Atlanta 10.0.0.1 64104 Texas 10.0.0.28 80 HTTP UDP 3357 +2010-12-01 23:39:06 Atlanta 10.0.0.1 54656 Texas 10.0.0.26 80 HTTP UDP 5275 +2010-12-01 23:39:07 Atlanta 10.0.0.1 52280 Texas 10.0.0.28 80 HTTP UDP 5952 +2010-12-01 23:39:08 Atlanta 10.0.0.1 63070 Texas 10.0.0.30 80 HTTP UDP 4176 +2010-12-01 23:39:09 Atlanta 10.0.0.1 54200 Texas 10.0.0.25 80 HTTP UDP 4410 +2010-12-01 23:39:10 Atlanta 10.0.0.1 50446 Texas 10.0.0.27 80 HTTP UDP 3545 +2010-12-01 23:39:11 Atlanta 10.0.0.1 61205 Texas 10.0.0.27 80 HTTP UDP 3099 +2010-12-01 23:39:12 Atlanta 10.0.0.1 58806 Texas 10.0.0.27 80 HTTP UDP 4886 +2010-12-01 23:39:13 Atlanta 10.0.0.1 55694 Texas 10.0.0.25 80 HTTP UDP 2909 +2010-12-01 23:39:14 Atlanta 10.0.0.1 61869 Texas 10.0.0.29 80 HTTP UDP 4241 +2010-12-01 23:39:15 Atlanta 10.0.0.1 56948 Texas 10.0.0.28 80 HTTP UDP 4189 +2010-12-01 23:39:16 Atlanta 10.0.0.1 59802 Texas 10.0.0.29 80 HTTP UDP 2946 +2010-12-01 23:39:17 Atlanta 10.0.0.1 55377 Texas 10.0.0.27 80 HTTP UDP 2860 +2010-12-01 23:39:18 Atlanta 10.0.0.1 53269 Texas 10.0.0.27 80 HTTP UDP 3549 +2010-12-01 23:39:19 Atlanta 10.0.0.1 55493 Texas 10.0.0.27 80 HTTP UDP 2819 +2010-12-01 23:39:20 Atlanta 10.0.0.1 61797 Texas 10.0.0.30 80 HTTP UDP 4352 +2010-12-01 23:39:21 Atlanta 10.0.0.1 57746 Texas 10.0.0.29 80 HTTP UDP 3806 +2010-12-01 23:39:22 Atlanta 10.0.0.1 53740 Texas 10.0.0.26 80 HTTP UDP 5096 +2010-12-01 23:39:23 Atlanta 10.0.0.1 55573 Texas 10.0.0.26 80 HTTP UDP 2975 +2010-12-01 23:39:24 Atlanta 10.0.0.1 63946 Texas 10.0.0.27 80 HTTP UDP 2579 +2010-12-01 23:39:25 Atlanta 10.0.0.1 49594 Texas 10.0.0.27 80 HTTP UDP 5043 +2010-12-01 23:39:26 Atlanta 10.0.0.1 52072 Texas 10.0.0.27 80 HTTP UDP 3136 +2010-12-01 23:39:27 Atlanta 10.0.0.1 59792 Texas 10.0.0.28 80 HTTP UDP 2813 +2010-12-01 23:39:28 Atlanta 10.0.0.1 49961 Texas 10.0.0.30 80 HTTP UDP 5327 +2010-12-01 23:39:29 Atlanta 10.0.0.1 54799 Texas 10.0.0.26 80 HTTP UDP 4794 +2010-12-01 23:39:30 Atlanta 10.0.0.1 55229 Texas 10.0.0.30 80 HTTP UDP 4410 +2010-12-01 23:39:31 Atlanta 10.0.0.1 50377 Texas 10.0.0.25 80 HTTP UDP 4488 +2010-12-01 23:39:32 Atlanta 10.0.0.1 58402 Texas 10.0.0.26 80 HTTP UDP 4327 +2010-12-01 23:39:33 Atlanta 10.0.0.1 57448 Texas 10.0.0.25 80 HTTP UDP 3959 +2010-12-01 23:39:34 Atlanta 10.0.0.1 61971 Texas 10.0.0.27 80 HTTP UDP 4195 +2010-12-01 23:39:35 Atlanta 10.0.0.1 64549 Texas 10.0.0.26 80 HTTP UDP 5837 +2010-12-01 23:39:36 Atlanta 10.0.0.1 60418 Texas 10.0.0.27 80 HTTP UDP 2026 +2010-12-01 23:39:37 Atlanta 10.0.0.1 58011 Texas 10.0.0.28 80 HTTP UDP 3183 +2010-12-01 23:39:38 Atlanta 10.0.0.1 57275 Texas 10.0.0.28 80 HTTP UDP 3079 +2010-12-01 23:39:39 Atlanta 10.0.0.1 57474 Texas 10.0.0.25 80 HTTP UDP 2012 +2010-12-01 23:39:40 Atlanta 10.0.0.1 63540 Texas 10.0.0.25 80 HTTP UDP 2294 +2010-12-01 23:39:41 Atlanta 10.0.0.1 54792 Texas 10.0.0.25 80 HTTP UDP 3764 +2010-12-01 23:39:42 Atlanta 10.0.0.1 56423 Texas 10.0.0.29 80 HTTP UDP 3607 +2010-12-01 23:39:43 Atlanta 10.0.0.1 56818 Texas 10.0.0.28 80 HTTP UDP 3431 +2010-12-01 23:39:44 Atlanta 10.0.0.1 52644 Texas 10.0.0.25 80 HTTP UDP 5696 +2010-12-01 23:39:45 Atlanta 10.0.0.1 50085 Texas 10.0.0.26 80 HTTP UDP 3855 +2010-12-01 23:39:46 Atlanta 10.0.0.1 56744 Texas 10.0.0.29 80 HTTP UDP 4449 +2010-12-01 23:39:47 Atlanta 10.0.0.1 58409 Texas 10.0.0.30 80 HTTP UDP 4041 +2010-12-01 23:39:48 Atlanta 10.0.0.1 56432 Texas 10.0.0.25 80 HTTP UDP 4903 +2010-12-01 23:39:49 Atlanta 10.0.0.1 63056 Texas 10.0.0.30 80 HTTP UDP 2102 +2010-12-01 23:39:50 Atlanta 10.0.0.1 57927 Texas 10.0.0.26 80 HTTP UDP 4424 +2010-12-01 23:39:51 Atlanta 10.0.0.1 52737 Texas 10.0.0.30 80 HTTP UDP 4103 +2010-12-01 23:39:52 Atlanta 10.0.0.1 62793 Texas 10.0.0.27 80 HTTP UDP 5501 +2010-12-01 23:39:53 Atlanta 10.0.0.1 55384 Texas 10.0.0.27 80 HTTP UDP 2366 +2010-12-01 23:39:54 Atlanta 10.0.0.1 55460 Texas 10.0.0.29 80 HTTP UDP 2298 +2010-12-01 23:39:55 Atlanta 10.0.0.1 63462 Texas 10.0.0.25 80 HTTP UDP 2523 +2010-12-01 23:39:56 Atlanta 10.0.0.1 64307 Texas 10.0.0.26 80 HTTP UDP 4948 +2010-12-01 23:39:57 Atlanta 10.0.0.1 55910 Texas 10.0.0.28 80 HTTP UDP 3028 +2010-12-01 23:39:58 Atlanta 10.0.0.1 51825 Texas 10.0.0.27 80 HTTP UDP 4600 +2010-12-01 23:39:59 Atlanta 10.0.0.1 55117 Texas 10.0.0.30 80 HTTP UDP 2924 +2010-12-01 23:40:00 Atlanta 10.0.0.1 53015 Texas 10.0.0.30 80 HTTP UDP 3230 +2010-12-01 23:40:01 Atlanta 10.0.0.1 62776 Texas 10.0.0.30 80 HTTP UDP 3936 +2010-12-01 23:40:02 Atlanta 10.0.0.1 56935 Texas 10.0.0.29 80 HTTP UDP 4013 +2010-12-01 23:40:03 Atlanta 10.0.0.1 60385 Texas 10.0.0.25 80 HTTP UDP 5035 +2010-12-01 23:40:04 Atlanta 10.0.0.1 53048 Texas 10.0.0.27 80 HTTP UDP 5712 +2010-12-01 23:40:05 Atlanta 10.0.0.1 61159 Texas 10.0.0.29 80 HTTP UDP 3160 +2010-12-01 23:40:06 Atlanta 10.0.0.1 61929 Texas 10.0.0.26 80 HTTP UDP 5162 +2010-12-01 23:40:07 Atlanta 10.0.0.1 49412 Texas 10.0.0.25 80 HTTP UDP 4032 +2010-12-01 23:40:08 Atlanta 10.0.0.1 50846 Texas 10.0.0.30 80 HTTP UDP 4717 +2010-12-01 23:40:09 Atlanta 10.0.0.1 50735 Texas 10.0.0.28 80 HTTP UDP 4377 +2010-12-01 23:40:10 Atlanta 10.0.0.1 60348 Texas 10.0.0.25 80 HTTP UDP 2818 +2010-12-01 23:40:11 Atlanta 10.0.0.1 49659 Texas 10.0.0.29 80 HTTP UDP 4921 +2010-12-01 23:40:12 Atlanta 10.0.0.1 52530 Texas 10.0.0.27 80 HTTP UDP 4253 +2010-12-01 23:40:13 Atlanta 10.0.0.1 59970 Texas 10.0.0.25 80 HTTP UDP 5620 +2010-12-01 23:40:14 Atlanta 10.0.0.1 53529 Texas 10.0.0.26 80 HTTP UDP 2674 +2010-12-01 23:40:15 Atlanta 10.0.0.1 55440 Texas 10.0.0.29 80 HTTP UDP 3125 +2010-12-01 23:40:16 Atlanta 10.0.0.1 60416 Texas 10.0.0.25 80 HTTP UDP 4576 +2010-12-01 23:40:17 Atlanta 10.0.0.1 52876 Texas 10.0.0.27 80 HTTP UDP 2311 +2010-12-01 23:40:18 Atlanta 10.0.0.1 61638 Texas 10.0.0.29 80 HTTP UDP 4653 +2010-12-01 23:40:19 Atlanta 10.0.0.1 60908 Texas 10.0.0.27 80 HTTP UDP 4705 +2010-12-01 23:40:20 Atlanta 10.0.0.1 50048 Texas 10.0.0.30 80 HTTP UDP 4359 +2010-12-01 23:40:21 Atlanta 10.0.0.1 61644 Texas 10.0.0.25 80 HTTP UDP 3828 +2010-12-01 23:40:22 Atlanta 10.0.0.1 64474 Texas 10.0.0.27 80 HTTP UDP 2630 +2010-12-01 23:40:23 Atlanta 10.0.0.1 50144 Texas 10.0.0.25 80 HTTP UDP 3084 +2010-12-01 23:40:24 Atlanta 10.0.0.1 63203 Texas 10.0.0.30 80 HTTP UDP 3325 +2010-12-01 23:40:25 Atlanta 10.0.0.1 58422 Texas 10.0.0.29 80 HTTP UDP 4393 +2010-12-01 23:40:26 Atlanta 10.0.0.1 57657 Texas 10.0.0.27 80 HTTP UDP 5422 +2010-12-01 23:40:27 Atlanta 10.0.0.1 55686 Texas 10.0.0.28 80 HTTP UDP 5806 +2010-12-01 23:40:28 Atlanta 10.0.0.1 49442 Texas 10.0.0.29 80 HTTP UDP 5284 +2010-12-01 23:40:29 Atlanta 10.0.0.1 54122 Texas 10.0.0.25 80 HTTP UDP 2462 +2010-12-01 23:40:30 Atlanta 10.0.0.1 62719 Texas 10.0.0.28 80 HTTP UDP 4701 +2010-12-01 23:40:31 Atlanta 10.0.0.1 57410 Texas 10.0.0.30 80 HTTP UDP 5689 +2010-12-01 23:40:32 Atlanta 10.0.0.1 58709 Texas 10.0.0.29 80 HTTP UDP 4410 +2010-12-01 23:40:33 Atlanta 10.0.0.1 61866 Texas 10.0.0.27 80 HTTP UDP 4653 +2010-12-01 23:40:34 Atlanta 10.0.0.1 53300 Texas 10.0.0.29 80 HTTP UDP 5841 +2010-12-01 23:40:35 Atlanta 10.0.0.1 53975 Texas 10.0.0.27 80 HTTP UDP 2627 +2010-12-01 23:40:36 Atlanta 10.0.0.1 63652 Texas 10.0.0.28 80 HTTP UDP 4777 +2010-12-01 23:40:37 Atlanta 10.0.0.1 58151 Texas 10.0.0.29 80 HTTP UDP 4229 +2010-12-01 23:40:38 Atlanta 10.0.0.1 62946 Texas 10.0.0.28 80 HTTP UDP 5369 +2010-12-01 23:40:39 Atlanta 10.0.0.1 64329 Texas 10.0.0.26 80 HTTP UDP 4046 +2010-12-01 23:40:40 Atlanta 10.0.0.1 57744 Texas 10.0.0.28 80 HTTP UDP 3875 +2010-12-01 23:40:41 Atlanta 10.0.0.1 54857 Texas 10.0.0.26 80 HTTP UDP 5918 +2010-12-01 23:40:42 Atlanta 10.0.0.1 53658 Texas 10.0.0.29 80 HTTP UDP 4636 +2010-12-01 23:40:43 Atlanta 10.0.0.1 59487 Texas 10.0.0.28 80 HTTP UDP 4792 +2010-12-01 23:40:44 Atlanta 10.0.0.1 54436 Texas 10.0.0.28 80 HTTP UDP 4407 +2010-12-01 23:40:45 Atlanta 10.0.0.1 49818 Texas 10.0.0.25 80 HTTP UDP 3171 +2010-12-01 23:40:46 Atlanta 10.0.0.1 54310 Texas 10.0.0.28 80 HTTP UDP 2311 +2010-12-01 23:40:47 Atlanta 10.0.0.1 63576 Texas 10.0.0.26 80 HTTP UDP 2813 +2010-12-01 23:40:48 Atlanta 10.0.0.1 63907 Texas 10.0.0.29 80 HTTP UDP 4058 +2010-12-01 23:40:49 Atlanta 10.0.0.1 62740 Texas 10.0.0.29 80 HTTP UDP 5952 +2010-12-01 23:40:50 Atlanta 10.0.0.1 62979 Texas 10.0.0.25 80 HTTP UDP 5605 +2010-12-01 23:40:51 Atlanta 10.0.0.1 56626 Texas 10.0.0.25 80 HTTP UDP 5593 +2010-12-01 23:40:52 Atlanta 10.0.0.1 62456 Texas 10.0.0.29 80 HTTP UDP 4229 +2010-12-01 23:40:53 Atlanta 10.0.0.1 49980 Texas 10.0.0.29 80 HTTP UDP 2462 +2010-12-01 23:40:54 Atlanta 10.0.0.1 64983 Texas 10.0.0.27 80 HTTP UDP 5256 +2010-12-01 23:40:55 Atlanta 10.0.0.1 52160 Texas 10.0.0.27 80 HTTP UDP 5020 +2010-12-01 23:40:56 Atlanta 10.0.0.1 59949 Texas 10.0.0.29 80 HTTP UDP 5628 +2010-12-01 23:40:57 Atlanta 10.0.0.1 58557 Texas 10.0.0.28 80 HTTP UDP 4925 +2010-12-01 23:40:58 Atlanta 10.0.0.1 57432 Texas 10.0.0.26 80 HTTP UDP 5392 +2010-12-01 23:40:59 Atlanta 10.0.0.1 61341 Texas 10.0.0.25 80 HTTP UDP 3216 +2010-12-01 23:41:00 Atlanta 10.0.0.1 55046 Texas 10.0.0.25 80 HTTP UDP 4886 +2010-12-01 23:41:01 Atlanta 10.0.0.1 61507 Texas 10.0.0.27 80 HTTP UDP 4744 +2010-12-01 23:41:02 Atlanta 10.0.0.1 57035 Texas 10.0.0.30 80 HTTP UDP 3196 +2010-12-01 23:41:03 Atlanta 10.0.0.1 57380 Texas 10.0.0.25 80 HTTP UDP 4955 +2010-12-01 23:41:04 Atlanta 10.0.0.1 50584 Texas 10.0.0.26 80 HTTP UDP 5673 +2010-12-01 23:41:05 Atlanta 10.0.0.1 61576 Texas 10.0.0.27 80 HTTP UDP 2663 +2010-12-01 23:41:06 Atlanta 10.0.0.1 50040 Texas 10.0.0.28 80 HTTP UDP 4696 +2010-12-01 23:41:07 Atlanta 10.0.0.1 52323 Texas 10.0.0.26 80 HTTP UDP 5925 +2010-12-01 23:41:08 Atlanta 10.0.0.1 64639 Texas 10.0.0.27 80 HTTP UDP 3345 +2010-12-01 23:41:09 Atlanta 10.0.0.1 62318 Texas 10.0.0.28 80 HTTP UDP 3122 +2010-12-01 23:41:10 Atlanta 10.0.0.1 51440 Texas 10.0.0.28 80 HTTP UDP 2580 +2010-12-01 23:41:11 Atlanta 10.0.0.1 64665 Texas 10.0.0.25 80 HTTP UDP 2114 +2010-12-01 23:41:12 Atlanta 10.0.0.1 62686 Texas 10.0.0.27 80 HTTP UDP 4701 +2010-12-01 23:41:13 Atlanta 10.0.0.1 64363 Texas 10.0.0.26 80 HTTP UDP 4414 +2010-12-01 23:41:14 Atlanta 10.0.0.1 51157 Texas 10.0.0.27 80 HTTP UDP 3669 +2010-12-01 23:41:15 Atlanta 10.0.0.1 57230 Texas 10.0.0.27 80 HTTP UDP 3032 +2010-12-01 23:41:16 Atlanta 10.0.0.1 60539 Texas 10.0.0.27 80 HTTP UDP 5700 +2010-12-01 23:41:17 Atlanta 10.0.0.1 52371 Texas 10.0.0.25 80 HTTP UDP 2217 +2010-12-01 23:41:18 Atlanta 10.0.0.1 58467 Texas 10.0.0.26 80 HTTP UDP 4921 +2010-12-01 23:41:19 Atlanta 10.0.0.1 54953 Texas 10.0.0.29 80 HTTP UDP 3947 +2010-12-01 23:41:20 Atlanta 10.0.0.1 50799 Texas 10.0.0.26 80 HTTP UDP 4369 +2010-12-01 23:41:21 Atlanta 10.0.0.1 58024 Texas 10.0.0.29 80 HTTP UDP 3224 +2010-12-01 23:41:22 Atlanta 10.0.0.1 55067 Texas 10.0.0.27 80 HTTP UDP 3758 +2010-12-01 23:41:23 Atlanta 10.0.0.1 52753 Texas 10.0.0.30 80 HTTP UDP 4565 +2010-12-01 23:41:24 Atlanta 10.0.0.1 50001 Texas 10.0.0.30 80 HTTP UDP 5355 +2010-12-01 23:41:25 Atlanta 10.0.0.1 50125 Texas 10.0.0.29 80 HTTP UDP 2934 +2010-12-01 23:41:26 Atlanta 10.0.0.1 51489 Texas 10.0.0.29 80 HTTP UDP 4395 +2010-12-01 23:41:27 Atlanta 10.0.0.1 52578 Texas 10.0.0.26 80 HTTP UDP 3793 +2010-12-01 23:41:28 Atlanta 10.0.0.1 53525 Texas 10.0.0.29 80 HTTP UDP 3634 +2010-12-01 23:41:29 Atlanta 10.0.0.1 58823 Texas 10.0.0.27 80 HTTP UDP 2380 +2010-12-01 23:41:30 Atlanta 10.0.0.1 61372 Texas 10.0.0.25 80 HTTP UDP 4501 +2010-12-01 23:41:31 Atlanta 10.0.0.1 55457 Texas 10.0.0.29 80 HTTP UDP 5866 +2010-12-01 23:41:32 Atlanta 10.0.0.1 56889 Texas 10.0.0.29 80 HTTP UDP 3136 +2010-12-01 23:41:33 Atlanta 10.0.0.1 59077 Texas 10.0.0.27 80 HTTP UDP 2587 +2010-12-01 23:41:34 Atlanta 10.0.0.1 57626 Texas 10.0.0.28 80 HTTP UDP 3445 +2010-12-01 23:41:35 Atlanta 10.0.0.1 51897 Texas 10.0.0.29 80 HTTP UDP 3499 +2010-12-01 23:41:36 Atlanta 10.0.0.1 56461 Texas 10.0.0.26 80 HTTP UDP 3618 +2010-12-01 23:41:37 Atlanta 10.0.0.1 59654 Texas 10.0.0.28 80 HTTP UDP 4179 +2010-12-01 23:41:38 Atlanta 10.0.0.1 59693 Texas 10.0.0.30 80 HTTP UDP 3174 +2010-12-01 23:41:39 Atlanta 10.0.0.1 62581 Texas 10.0.0.30 80 HTTP UDP 4750 +2010-12-01 23:41:40 Atlanta 10.0.0.1 54529 Texas 10.0.0.28 80 HTTP UDP 2690 +2010-12-01 23:41:41 Atlanta 10.0.0.1 49998 Texas 10.0.0.28 80 HTTP UDP 4969 +2010-12-01 23:41:42 Atlanta 10.0.0.1 58948 Texas 10.0.0.27 80 HTTP UDP 2652 +2010-12-01 23:41:43 Atlanta 10.0.0.1 60664 Texas 10.0.0.26 80 HTTP UDP 3107 +2010-12-01 23:41:44 Atlanta 10.0.0.1 55713 Texas 10.0.0.27 80 HTTP UDP 3144 +2010-12-01 23:41:45 Atlanta 10.0.0.1 64382 Texas 10.0.0.26 80 HTTP UDP 2910 +2010-12-01 23:41:46 Atlanta 10.0.0.1 58998 Texas 10.0.0.30 80 HTTP UDP 3568 +2010-12-01 23:41:47 Atlanta 10.0.0.1 52977 Texas 10.0.0.27 80 HTTP UDP 3042 +2010-12-01 23:41:48 Atlanta 10.0.0.1 54765 Texas 10.0.0.28 80 HTTP UDP 5662 +2010-12-01 23:41:49 Atlanta 10.0.0.1 56147 Texas 10.0.0.30 80 HTTP UDP 5256 +2010-12-01 23:41:50 Atlanta 10.0.0.1 65344 Texas 10.0.0.27 80 HTTP UDP 4636 +2010-12-01 23:41:51 Atlanta 10.0.0.1 55907 Texas 10.0.0.25 80 HTTP UDP 4006 +2010-12-01 23:41:52 Atlanta 10.0.0.1 60858 Texas 10.0.0.27 80 HTTP UDP 4410 +2010-12-01 23:41:53 Atlanta 10.0.0.1 51672 Texas 10.0.0.27 80 HTTP UDP 5377 +2010-12-01 23:41:54 Atlanta 10.0.0.1 51776 Texas 10.0.0.28 80 HTTP UDP 2249 +2010-12-01 23:41:55 Atlanta 10.0.0.1 52962 Texas 10.0.0.25 80 HTTP UDP 3326 +2010-12-01 23:41:56 Atlanta 10.0.0.1 52680 Texas 10.0.0.29 80 HTTP UDP 5849 +2010-12-01 23:41:57 Atlanta 10.0.0.1 62691 Texas 10.0.0.25 80 HTTP UDP 2312 +2010-12-01 23:41:58 Atlanta 10.0.0.1 60393 Texas 10.0.0.28 80 HTTP UDP 2361 +2010-12-01 23:41:59 Atlanta 10.0.0.1 54506 Texas 10.0.0.29 80 HTTP UDP 3183 +2010-12-01 23:42:00 Atlanta 10.0.0.1 52206 Texas 10.0.0.29 80 HTTP UDP 5217 +2010-12-01 23:42:01 Atlanta 10.0.0.1 51666 Texas 10.0.0.29 80 HTTP UDP 3826 +2010-12-01 23:42:02 Atlanta 10.0.0.1 55136 Texas 10.0.0.27 80 HTTP UDP 2553 +2010-12-01 23:42:03 Atlanta 10.0.0.1 55873 Texas 10.0.0.26 80 HTTP UDP 2313 +2010-12-01 23:42:04 Atlanta 10.0.0.1 56618 Texas 10.0.0.27 80 HTTP UDP 3864 +2010-12-01 23:42:05 Atlanta 10.0.0.1 59187 Texas 10.0.0.25 80 HTTP UDP 2997 +2010-12-01 23:42:06 Atlanta 10.0.0.1 58910 Texas 10.0.0.27 80 HTTP UDP 2579 +2010-12-01 23:42:07 Atlanta 10.0.0.1 55178 Texas 10.0.0.29 80 HTTP UDP 2294 +2010-12-01 23:42:08 Atlanta 10.0.0.1 60834 Texas 10.0.0.25 80 HTTP UDP 3398 +2010-12-01 23:42:09 Atlanta 10.0.0.1 53272 Texas 10.0.0.29 80 HTTP UDP 2979 +2010-12-01 23:42:10 Atlanta 10.0.0.1 51742 Texas 10.0.0.25 80 HTTP UDP 5750 +2010-12-01 23:42:11 Atlanta 10.0.0.1 62970 Texas 10.0.0.29 80 HTTP UDP 2559 +2010-12-01 23:42:12 Atlanta 10.0.0.1 64370 Texas 10.0.0.27 80 HTTP UDP 3079 +2010-12-01 23:42:13 Atlanta 10.0.0.1 58248 Texas 10.0.0.28 80 HTTP UDP 4744 +2010-12-01 23:42:14 Atlanta 10.0.0.1 53691 Texas 10.0.0.27 80 HTTP UDP 2652 +2010-12-01 23:42:15 Atlanta 10.0.0.1 62752 Texas 10.0.0.28 80 HTTP UDP 2192 +2010-12-01 23:42:16 Atlanta 10.0.0.1 49447 Texas 10.0.0.28 80 HTTP UDP 3940 +2010-12-01 23:42:17 Atlanta 10.0.0.1 51714 Texas 10.0.0.30 80 HTTP UDP 5593 +2010-12-01 23:42:18 Atlanta 10.0.0.1 62068 Texas 10.0.0.28 80 HTTP UDP 5684 +2010-12-01 23:42:19 Atlanta 10.0.0.1 57316 Texas 10.0.0.30 80 HTTP UDP 4502 +2010-12-01 23:42:20 Atlanta 10.0.0.1 61675 Texas 10.0.0.29 80 HTTP UDP 2924 +2010-12-01 23:42:21 Atlanta 10.0.0.1 49575 Texas 10.0.0.29 80 HTTP UDP 4397 +2010-12-01 23:42:22 Atlanta 10.0.0.1 63447 Texas 10.0.0.27 80 HTTP UDP 5856 +2010-12-01 23:42:23 Atlanta 10.0.0.1 50535 Texas 10.0.0.25 80 HTTP UDP 2002 +2010-12-01 23:42:24 Atlanta 10.0.0.1 62232 Texas 10.0.0.30 80 HTTP UDP 3107 +2010-12-01 23:42:25 Atlanta 10.0.0.1 54368 Texas 10.0.0.26 80 HTTP UDP 4358 +2010-12-01 23:42:26 Atlanta 10.0.0.1 59660 Texas 10.0.0.26 80 HTTP UDP 5954 +2010-12-01 23:42:27 Atlanta 10.0.0.1 53802 Texas 10.0.0.25 80 HTTP UDP 5341 +2010-12-01 23:42:28 Atlanta 10.0.0.1 50745 Texas 10.0.0.30 80 HTTP UDP 5807 +2010-12-01 23:42:29 Atlanta 10.0.0.1 58716 Texas 10.0.0.29 80 HTTP UDP 5696 +2010-12-01 23:42:30 Atlanta 10.0.0.1 60358 Texas 10.0.0.30 80 HTTP UDP 2361 +2010-12-01 23:42:31 Atlanta 10.0.0.1 49677 Texas 10.0.0.27 80 HTTP UDP 3433 +2010-12-01 23:42:32 Atlanta 10.0.0.1 54984 Texas 10.0.0.25 80 HTTP UDP 3228 +2010-12-01 23:42:33 Atlanta 10.0.0.1 58028 Texas 10.0.0.29 80 HTTP UDP 4397 +2010-12-01 23:42:34 Atlanta 10.0.0.1 63595 Texas 10.0.0.26 80 HTTP UDP 3400 +2010-12-01 23:42:35 Atlanta 10.0.0.1 58026 Texas 10.0.0.26 80 HTTP UDP 2102 +2010-12-01 23:42:36 Atlanta 10.0.0.1 64357 Texas 10.0.0.27 80 HTTP UDP 4191 +2010-12-01 23:42:37 Atlanta 10.0.0.1 56639 Texas 10.0.0.29 80 HTTP UDP 5541 +2010-12-01 23:42:38 Atlanta 10.0.0.1 64812 Texas 10.0.0.27 80 HTTP UDP 5811 +2010-12-01 23:42:39 Atlanta 10.0.0.1 49416 Texas 10.0.0.25 80 HTTP UDP 3826 +2010-12-01 23:42:40 Atlanta 10.0.0.1 58140 Texas 10.0.0.28 80 HTTP UDP 3995 +2010-12-01 23:42:41 Atlanta 10.0.0.1 56243 Texas 10.0.0.30 80 HTTP UDP 3522 +2010-12-01 23:42:42 Atlanta 10.0.0.1 60350 Texas 10.0.0.25 80 HTTP UDP 4581 +2010-12-01 23:42:43 Atlanta 10.0.0.1 58146 Texas 10.0.0.28 80 HTTP UDP 5558 +2010-12-01 23:42:44 Atlanta 10.0.0.1 62187 Texas 10.0.0.29 80 HTTP UDP 5464 +2010-12-01 23:42:45 Atlanta 10.0.0.1 52992 Texas 10.0.0.28 80 HTTP UDP 2133 +2010-12-01 23:42:46 Atlanta 10.0.0.1 56706 Texas 10.0.0.30 80 HTTP UDP 5301 +2010-12-01 23:42:47 Atlanta 10.0.0.1 52185 Texas 10.0.0.30 80 HTTP UDP 2318 +2010-12-01 23:42:48 Atlanta 10.0.0.1 61711 Texas 10.0.0.25 80 HTTP UDP 4191 +2010-12-01 23:42:49 Atlanta 10.0.0.1 56900 Texas 10.0.0.28 80 HTTP UDP 4041 +2010-12-01 23:42:50 Atlanta 10.0.0.1 56614 Texas 10.0.0.28 80 HTTP UDP 2627 +2010-12-01 23:42:51 Atlanta 10.0.0.1 63729 Texas 10.0.0.28 80 HTTP UDP 2579 +2010-12-01 23:42:52 Atlanta 10.0.0.1 63854 Texas 10.0.0.27 80 HTTP UDP 2598 +2010-12-01 23:42:53 Atlanta 10.0.0.1 50598 Texas 10.0.0.29 80 HTTP UDP 3864 +2010-12-01 23:42:54 Atlanta 10.0.0.1 60387 Texas 10.0.0.27 80 HTTP UDP 3934 +2010-12-01 23:42:55 Atlanta 10.0.0.1 60534 Texas 10.0.0.25 80 HTTP UDP 2994 +2010-12-01 23:42:56 Atlanta 10.0.0.1 50879 Texas 10.0.0.25 80 HTTP UDP 3861 +2010-12-01 23:42:57 Atlanta 10.0.0.1 63765 Texas 10.0.0.25 80 HTTP UDP 2257 +2010-12-01 23:42:58 Atlanta 10.0.0.1 49666 Texas 10.0.0.30 80 HTTP UDP 5015 +2010-12-01 23:42:59 Atlanta 10.0.0.1 50529 Texas 10.0.0.25 80 HTTP UDP 5849 +2010-12-01 23:43:00 Atlanta 10.0.0.1 60701 Texas 10.0.0.29 80 HTTP UDP 4566 +2010-12-01 23:43:01 Atlanta 10.0.0.1 55059 Texas 10.0.0.28 80 HTTP UDP 2553 +2010-12-01 23:43:02 Atlanta 10.0.0.1 49405 Texas 10.0.0.27 80 HTTP UDP 2015 +2010-12-01 23:43:03 Atlanta 10.0.0.1 56252 Texas 10.0.0.28 80 HTTP UDP 5850 +2010-12-01 23:43:04 Atlanta 10.0.0.1 49253 Texas 10.0.0.25 80 HTTP UDP 4751 +2010-12-01 23:43:05 Atlanta 10.0.0.1 55603 Texas 10.0.0.28 80 HTTP UDP 2172 +2010-12-01 23:43:06 Atlanta 10.0.0.1 58299 Texas 10.0.0.29 80 HTTP UDP 5225 +2010-12-01 23:43:07 Atlanta 10.0.0.1 56518 Texas 10.0.0.28 80 HTTP UDP 2361 +2010-12-01 23:43:08 Atlanta 10.0.0.1 63017 Texas 10.0.0.26 80 HTTP UDP 5774 +2010-12-01 23:43:09 Atlanta 10.0.0.1 64387 Texas 10.0.0.25 80 HTTP UDP 5828 +2010-12-01 23:43:10 Atlanta 10.0.0.1 64691 Texas 10.0.0.28 80 HTTP UDP 4922 +2010-12-01 23:43:11 Atlanta 10.0.0.1 57070 Texas 10.0.0.29 80 HTTP UDP 5016 +2010-12-01 23:43:12 Atlanta 10.0.0.1 52870 Texas 10.0.0.30 80 HTTP UDP 2311 +2010-12-01 23:43:13 Atlanta 10.0.0.1 64998 Texas 10.0.0.25 80 HTTP UDP 3066 +2010-12-01 23:43:14 Atlanta 10.0.0.1 50228 Texas 10.0.0.27 80 HTTP UDP 5750 +2010-12-01 23:43:15 Atlanta 10.0.0.1 54181 Texas 10.0.0.30 80 HTTP UDP 3010 +2010-12-01 23:43:16 Atlanta 10.0.0.1 53624 Texas 10.0.0.28 80 HTTP UDP 3602 +2010-12-01 23:43:17 Atlanta 10.0.0.1 56613 Texas 10.0.0.26 80 HTTP UDP 5484 +2010-12-01 23:43:18 Atlanta 10.0.0.1 56747 Texas 10.0.0.29 80 HTTP UDP 3758 +2010-12-01 23:43:19 Atlanta 10.0.0.1 63462 Texas 10.0.0.30 80 HTTP UDP 4931 +2010-12-01 23:43:20 Atlanta 10.0.0.1 53600 Texas 10.0.0.26 80 HTTP UDP 4093 +2010-12-01 23:43:21 Atlanta 10.0.0.1 61021 Texas 10.0.0.28 80 HTTP UDP 2026 +2010-12-01 23:43:22 Atlanta 10.0.0.1 50284 Texas 10.0.0.29 80 HTTP UDP 5285 +2010-12-01 23:43:23 Atlanta 10.0.0.1 55111 Texas 10.0.0.28 80 HTTP UDP 5617 +2010-12-01 23:43:24 Atlanta 10.0.0.1 63494 Texas 10.0.0.25 80 HTTP UDP 3501 +2010-12-01 23:43:25 Atlanta 10.0.0.1 58188 Texas 10.0.0.30 80 HTTP UDP 4229 +2010-12-01 23:43:26 Atlanta 10.0.0.1 56167 Texas 10.0.0.25 80 HTTP UDP 2577 +2010-12-01 23:43:27 Atlanta 10.0.0.1 60388 Texas 10.0.0.29 80 HTTP UDP 2380 +2010-12-01 23:43:28 Atlanta 10.0.0.1 60593 Texas 10.0.0.27 80 HTTP UDP 2579 +2010-12-01 23:43:29 Atlanta 10.0.0.1 59488 Texas 10.0.0.29 80 HTTP UDP 4933 +2010-12-01 23:43:30 Atlanta 10.0.0.1 51190 Texas 10.0.0.30 80 HTTP UDP 2015 +2010-12-01 23:43:31 Atlanta 10.0.0.1 51610 Texas 10.0.0.28 80 HTTP UDP 4561 +2010-12-01 23:43:32 Atlanta 10.0.0.1 60554 Texas 10.0.0.25 80 HTTP UDP 4211 +2010-12-01 23:43:33 Atlanta 10.0.0.1 61766 Texas 10.0.0.25 80 HTTP UDP 5035 +2010-12-01 23:43:34 Atlanta 10.0.0.1 61572 Texas 10.0.0.28 80 HTTP UDP 2217 +2010-12-01 23:43:35 Atlanta 10.0.0.1 54710 Texas 10.0.0.28 80 HTTP UDP 3216 +2010-12-01 23:43:36 Atlanta 10.0.0.1 62284 Texas 10.0.0.26 80 HTTP UDP 3433 +2010-12-01 23:43:37 Atlanta 10.0.0.1 61355 Texas 10.0.0.28 80 HTTP UDP 5506 +2010-12-01 23:43:38 Atlanta 10.0.0.1 55879 Texas 10.0.0.27 80 HTTP UDP 5556 +2010-12-01 23:43:39 Atlanta 10.0.0.1 51343 Texas 10.0.0.27 80 HTTP UDP 4369 +2010-12-01 23:43:40 Atlanta 10.0.0.1 61361 Texas 10.0.0.25 80 HTTP UDP 5217 +2010-12-01 23:43:41 Atlanta 10.0.0.1 63731 Texas 10.0.0.29 80 HTTP UDP 3171 +2010-12-01 23:43:42 Atlanta 10.0.0.1 61834 Texas 10.0.0.30 80 HTTP UDP 5162 +2010-12-01 23:43:43 Atlanta 10.0.0.1 51311 Texas 10.0.0.29 80 HTTP UDP 4925 +2010-12-01 23:43:44 Atlanta 10.0.0.1 63857 Texas 10.0.0.28 80 HTTP UDP 2551 +2010-12-01 23:43:45 Atlanta 10.0.0.1 62590 Texas 10.0.0.28 80 HTTP UDP 3431 +2010-12-01 23:43:46 Atlanta 10.0.0.1 61494 Texas 10.0.0.29 80 HTTP UDP 2422 +2010-12-01 23:43:47 Atlanta 10.0.0.1 63158 Texas 10.0.0.29 80 HTTP UDP 4399 +2010-12-01 23:43:48 Atlanta 10.0.0.1 63580 Texas 10.0.0.26 80 HTTP UDP 3138 +2010-12-01 23:43:49 Atlanta 10.0.0.1 57609 Texas 10.0.0.27 80 HTTP UDP 3743 +2010-12-01 23:43:50 Atlanta 10.0.0.1 62614 Texas 10.0.0.29 80 HTTP UDP 4496 +2010-12-01 23:43:51 Atlanta 10.0.0.1 63104 Texas 10.0.0.30 80 HTTP UDP 4501 +2010-12-01 23:43:52 Atlanta 10.0.0.1 59519 Texas 10.0.0.27 80 HTTP UDP 4424 +2010-12-01 23:43:53 Atlanta 10.0.0.1 57939 Texas 10.0.0.26 80 HTTP UDP 5309 +2010-12-01 23:43:54 Atlanta 10.0.0.1 52965 Texas 10.0.0.27 80 HTTP UDP 4414 +2010-12-01 23:43:55 Atlanta 10.0.0.1 61387 Texas 10.0.0.29 80 HTTP UDP 3114 +2010-12-01 23:43:56 Atlanta 10.0.0.1 52620 Texas 10.0.0.26 80 HTTP UDP 4258 +2010-12-01 23:43:57 Atlanta 10.0.0.1 51781 Texas 10.0.0.28 80 HTTP UDP 5506 +2010-12-01 23:43:58 Atlanta 10.0.0.1 60095 Texas 10.0.0.25 80 HTTP UDP 4088 +2010-12-01 23:43:59 Atlanta 10.0.0.1 64300 Texas 10.0.0.30 80 HTTP UDP 3764 +2010-12-01 23:44:00 Atlanta 10.0.0.1 54126 Texas 10.0.0.25 80 HTTP UDP 2867 +2010-12-01 23:44:01 Atlanta 10.0.0.1 62836 Texas 10.0.0.28 80 HTTP UDP 5570 +2010-12-01 23:44:02 Atlanta 10.0.0.1 56324 Texas 10.0.0.29 80 HTTP UDP 5511 +2010-12-01 23:44:03 Atlanta 10.0.0.1 65254 Texas 10.0.0.30 80 HTTP UDP 2617 +2010-12-01 23:44:04 Atlanta 10.0.0.1 62067 Texas 10.0.0.27 80 HTTP UDP 3067 +2010-12-01 23:44:05 Atlanta 10.0.0.1 54046 Texas 10.0.0.30 80 HTTP UDP 3317 +2010-12-01 23:44:06 Atlanta 10.0.0.1 58352 Texas 10.0.0.27 80 HTTP UDP 5347 +2010-12-01 23:44:07 Atlanta 10.0.0.1 55234 Texas 10.0.0.25 80 HTTP UDP 3198 +2010-12-01 23:44:08 Atlanta 10.0.0.1 59203 Texas 10.0.0.27 80 HTTP UDP 5217 +2010-12-01 23:44:09 Atlanta 10.0.0.1 54170 Texas 10.0.0.25 80 HTTP UDP 3562 +2010-12-01 23:44:10 Atlanta 10.0.0.1 58548 Texas 10.0.0.26 80 HTTP UDP 4948 +2010-12-01 23:44:11 Atlanta 10.0.0.1 52367 Texas 10.0.0.30 80 HTTP UDP 5605 +2010-12-01 23:44:12 Atlanta 10.0.0.1 57081 Texas 10.0.0.25 80 HTTP UDP 2640 +2010-12-01 23:44:13 Atlanta 10.0.0.1 62446 Texas 10.0.0.27 80 HTTP UDP 3918 +2010-12-01 23:44:14 Atlanta 10.0.0.1 57982 Texas 10.0.0.27 80 HTTP UDP 3421 +2010-12-01 23:44:15 Atlanta 10.0.0.1 65383 Texas 10.0.0.29 80 HTTP UDP 4886 +2010-12-01 23:44:16 Atlanta 10.0.0.1 55754 Texas 10.0.0.30 80 HTTP UDP 4395 +2010-12-01 23:44:17 Atlanta 10.0.0.1 49325 Texas 10.0.0.28 80 HTTP UDP 2694 +2010-12-01 23:44:18 Atlanta 10.0.0.1 64772 Texas 10.0.0.26 80 HTTP UDP 2684 +2010-12-01 23:44:19 Atlanta 10.0.0.1 58453 Texas 10.0.0.28 80 HTTP UDP 2378 +2010-12-01 23:44:20 Atlanta 10.0.0.1 63986 Texas 10.0.0.30 80 HTTP UDP 5710 +2010-12-01 23:44:21 Atlanta 10.0.0.1 53459 Texas 10.0.0.27 80 HTTP UDP 2997 +2010-12-01 23:44:22 Atlanta 10.0.0.1 58338 Texas 10.0.0.29 80 HTTP UDP 2689 +2010-12-01 23:44:23 Atlanta 10.0.0.1 59692 Texas 10.0.0.28 80 HTTP UDP 3900 +2010-12-01 23:44:24 Atlanta 10.0.0.1 51049 Texas 10.0.0.28 80 HTTP UDP 2996 +2010-12-01 23:44:25 Atlanta 10.0.0.1 52823 Texas 10.0.0.27 80 HTTP UDP 5171 +2010-12-01 23:44:26 Atlanta 10.0.0.1 63210 Texas 10.0.0.27 80 HTTP UDP 5998 +2010-12-01 23:44:27 Atlanta 10.0.0.1 64828 Texas 10.0.0.27 80 HTTP UDP 2979 +2010-12-01 23:44:28 Atlanta 10.0.0.1 50861 Texas 10.0.0.26 80 HTTP UDP 2235 +2010-12-01 23:44:29 Atlanta 10.0.0.1 60435 Texas 10.0.0.25 80 HTTP UDP 3419 +2010-12-01 23:44:30 Atlanta 10.0.0.1 61422 Texas 10.0.0.29 80 HTTP UDP 5016 +2010-12-01 23:44:31 Atlanta 10.0.0.1 57547 Texas 10.0.0.29 80 HTTP UDP 2860 +2010-12-01 23:44:32 Atlanta 10.0.0.1 62140 Texas 10.0.0.28 80 HTTP UDP 3544 +2010-12-01 23:44:33 Atlanta 10.0.0.1 58887 Texas 10.0.0.27 80 HTTP UDP 2221 +2010-12-01 23:44:34 Atlanta 10.0.0.1 58268 Texas 10.0.0.30 80 HTTP UDP 3042 +2010-12-01 23:44:35 Atlanta 10.0.0.1 50878 Texas 10.0.0.28 80 HTTP UDP 3146 +2010-12-01 23:44:36 Atlanta 10.0.0.1 61199 Texas 10.0.0.28 80 HTTP UDP 2114 +2010-12-01 23:44:37 Atlanta 10.0.0.1 54500 Texas 10.0.0.27 80 HTTP UDP 4566 +2010-12-01 23:44:38 Atlanta 10.0.0.1 57056 Texas 10.0.0.26 80 HTTP UDP 4777 +2010-12-01 23:44:39 Atlanta 10.0.0.1 52890 Texas 10.0.0.28 80 HTTP UDP 2813 +2010-12-01 23:44:40 Atlanta 10.0.0.1 65232 Texas 10.0.0.30 80 HTTP UDP 4414 +2010-12-01 23:44:41 Atlanta 10.0.0.1 64931 Texas 10.0.0.25 80 HTTP UDP 4899 +2010-12-01 23:44:42 Atlanta 10.0.0.1 60818 Texas 10.0.0.26 80 HTTP UDP 3016 +2010-12-01 23:44:43 Atlanta 10.0.0.1 50687 Texas 10.0.0.28 80 HTTP UDP 3511 +2010-12-01 23:44:44 Atlanta 10.0.0.1 64800 Texas 10.0.0.28 80 HTTP UDP 4817 +2010-12-01 23:44:45 Atlanta 10.0.0.1 54602 Texas 10.0.0.29 80 HTTP UDP 2332 +2010-12-01 23:44:46 Atlanta 10.0.0.1 51640 Texas 10.0.0.26 80 HTTP UDP 5275 +2010-12-01 23:44:47 Atlanta 10.0.0.1 57461 Texas 10.0.0.27 80 HTTP UDP 4356 +2010-12-01 23:44:48 Atlanta 10.0.0.1 62671 Texas 10.0.0.28 80 HTTP UDP 3445 +2010-12-01 23:44:49 Atlanta 10.0.0.1 64253 Texas 10.0.0.26 80 HTTP UDP 3855 +2010-12-01 23:44:50 Atlanta 10.0.0.1 56067 Texas 10.0.0.29 80 HTTP UDP 5350 +2010-12-01 23:44:51 Atlanta 10.0.0.1 53499 Texas 10.0.0.28 80 HTTP UDP 5128 +2010-12-01 23:44:52 Atlanta 10.0.0.1 49279 Texas 10.0.0.28 80 HTTP UDP 5392 +2010-12-01 23:44:53 Atlanta 10.0.0.1 55841 Texas 10.0.0.29 80 HTTP UDP 5347 +2010-12-01 23:44:54 Atlanta 10.0.0.1 64574 Texas 10.0.0.25 80 HTTP UDP 5673 +2010-12-01 23:44:55 Atlanta 10.0.0.1 53014 Texas 10.0.0.25 80 HTTP UDP 4920 +2010-12-01 23:44:56 Atlanta 10.0.0.1 61543 Texas 10.0.0.29 80 HTTP UDP 3131 +2010-12-01 23:44:57 Atlanta 10.0.0.1 51886 Texas 10.0.0.30 80 HTTP UDP 4064 +2010-12-01 23:44:58 Atlanta 10.0.0.1 57070 Texas 10.0.0.29 80 HTTP UDP 4945 +2010-12-01 23:44:59 Atlanta 10.0.0.1 60838 Texas 10.0.0.30 80 HTTP UDP 3019 +2010-12-01 23:45:00 Atlanta 10.0.0.1 51528 Texas 10.0.0.29 80 HTTP UDP 5255 +2010-12-01 23:45:01 Atlanta 10.0.0.1 61854 Texas 10.0.0.28 80 HTTP UDP 5710 +2010-12-01 23:45:02 Atlanta 10.0.0.1 62697 Texas 10.0.0.27 80 HTTP UDP 3347 +2010-12-01 23:45:03 Atlanta 10.0.0.1 49935 Texas 10.0.0.26 80 HTTP UDP 3268 +2010-12-01 23:45:04 Atlanta 10.0.0.1 58732 Texas 10.0.0.26 80 HTTP UDP 2954 +2010-12-01 23:45:05 Atlanta 10.0.0.1 61062 Texas 10.0.0.29 80 HTTP UDP 3466 +2010-12-01 23:45:06 Atlanta 10.0.0.1 59671 Texas 10.0.0.29 80 HTTP UDP 2135 +2010-12-01 23:45:07 Atlanta 10.0.0.1 54441 Texas 10.0.0.27 80 HTTP UDP 2656 +2010-12-01 23:45:08 Atlanta 10.0.0.1 61409 Texas 10.0.0.29 80 HTTP UDP 4567 +2010-12-01 23:45:09 Atlanta 10.0.0.1 52173 Texas 10.0.0.26 80 HTTP UDP 2972 +2010-12-01 23:45:10 Atlanta 10.0.0.1 59763 Texas 10.0.0.26 80 HTTP UDP 2172 +2010-12-01 23:45:11 Atlanta 10.0.0.1 51129 Texas 10.0.0.25 80 HTTP UDP 5527 +2010-12-01 23:45:12 Atlanta 10.0.0.1 53194 Texas 10.0.0.27 80 HTTP UDP 2813 +2010-12-01 23:45:13 Atlanta 10.0.0.1 57271 Texas 10.0.0.26 80 HTTP UDP 2003 +2010-12-01 23:45:14 Atlanta 10.0.0.1 63602 Texas 10.0.0.29 80 HTTP UDP 2580 +2010-12-01 23:45:15 Atlanta 10.0.0.1 56666 Texas 10.0.0.29 80 HTTP UDP 4440 +2010-12-01 23:45:16 Atlanta 10.0.0.1 58983 Texas 10.0.0.29 80 HTTP UDP 4744 +2010-12-01 23:45:17 Atlanta 10.0.0.1 62308 Texas 10.0.0.26 80 HTTP UDP 5125 +2010-12-01 23:45:18 Atlanta 10.0.0.1 49444 Texas 10.0.0.27 80 HTTP UDP 2603 +2010-12-01 23:45:19 Atlanta 10.0.0.1 64857 Texas 10.0.0.30 80 HTTP UDP 5389 +2010-12-01 23:45:20 Atlanta 10.0.0.1 60794 Texas 10.0.0.26 80 HTTP UDP 3970 +2010-12-01 23:45:21 Atlanta 10.0.0.1 51817 Texas 10.0.0.26 80 HTTP UDP 5673 +2010-12-01 23:45:22 Atlanta 10.0.0.1 58288 Texas 10.0.0.26 80 HTTP UDP 3017 +2010-12-01 23:45:23 Atlanta 10.0.0.1 65377 Texas 10.0.0.30 80 HTTP UDP 2940 +2010-12-01 23:45:24 Atlanta 10.0.0.1 61384 Texas 10.0.0.30 80 HTTP UDP 3618 +2010-12-01 23:45:25 Atlanta 10.0.0.1 57345 Texas 10.0.0.27 80 HTTP UDP 4627 +2010-12-01 23:45:26 Atlanta 10.0.0.1 65211 Texas 10.0.0.28 80 HTTP UDP 4238 +2010-12-01 23:45:27 Atlanta 10.0.0.1 63714 Texas 10.0.0.27 80 HTTP UDP 4040 +2010-12-01 23:45:28 Atlanta 10.0.0.1 50629 Texas 10.0.0.30 80 HTTP UDP 4705 +2010-12-01 23:45:29 Atlanta 10.0.0.1 59907 Texas 10.0.0.29 80 HTTP UDP 2409 +2010-12-01 23:45:30 Atlanta 10.0.0.1 56052 Texas 10.0.0.25 80 HTTP UDP 2012 +2010-12-01 23:45:31 Atlanta 10.0.0.1 50721 Texas 10.0.0.29 80 HTTP UDP 4683 +2010-12-01 23:45:32 Atlanta 10.0.0.1 53995 Texas 10.0.0.26 80 HTTP UDP 5336 +2010-12-01 23:45:33 Atlanta 10.0.0.1 60418 Texas 10.0.0.25 80 HTTP UDP 2380 +2010-12-01 23:45:34 Atlanta 10.0.0.1 61698 Texas 10.0.0.30 80 HTTP UDP 5145 +2010-12-01 23:45:35 Atlanta 10.0.0.1 50147 Texas 10.0.0.29 80 HTTP UDP 4824 +2010-12-01 23:45:36 Atlanta 10.0.0.1 63628 Texas 10.0.0.25 80 HTTP UDP 3052 +2010-12-01 23:45:37 Atlanta 10.0.0.1 60314 Texas 10.0.0.30 80 HTTP UDP 5603 +2010-12-01 23:45:38 Atlanta 10.0.0.1 63950 Texas 10.0.0.25 80 HTTP UDP 4328 +2010-12-01 23:45:39 Atlanta 10.0.0.1 53060 Texas 10.0.0.25 80 HTTP UDP 2447 +2010-12-01 23:45:40 Atlanta 10.0.0.1 55352 Texas 10.0.0.27 80 HTTP UDP 2947 +2010-12-01 23:45:41 Atlanta 10.0.0.1 61135 Texas 10.0.0.25 80 HTTP UDP 2562 +2010-12-01 23:45:42 Atlanta 10.0.0.1 58524 Texas 10.0.0.29 80 HTTP UDP 4777 +2010-12-01 23:45:43 Atlanta 10.0.0.1 55623 Texas 10.0.0.29 80 HTTP UDP 3806 +2010-12-01 23:45:44 Atlanta 10.0.0.1 54520 Texas 10.0.0.25 80 HTTP UDP 5867 +2010-12-01 23:45:45 Atlanta 10.0.0.1 58822 Texas 10.0.0.25 80 HTTP UDP 5545 +2010-12-01 23:45:46 Atlanta 10.0.0.1 62659 Texas 10.0.0.29 80 HTTP UDP 3935 +2010-12-01 23:45:47 Atlanta 10.0.0.1 63776 Texas 10.0.0.26 80 HTTP UDP 2956 +2010-12-01 23:45:48 Atlanta 10.0.0.1 62566 Texas 10.0.0.26 80 HTTP UDP 2810 +2010-12-01 23:45:49 Atlanta 10.0.0.1 65335 Texas 10.0.0.28 80 HTTP UDP 2498 +2010-12-01 23:45:50 Atlanta 10.0.0.1 65287 Texas 10.0.0.26 80 HTTP UDP 2580 +2010-12-01 23:45:51 Atlanta 10.0.0.1 64749 Texas 10.0.0.29 80 HTTP UDP 4728 +2010-12-01 23:45:52 Atlanta 10.0.0.1 52572 Texas 10.0.0.28 80 HTTP UDP 3538 +2010-12-01 23:45:53 Atlanta 10.0.0.1 63924 Texas 10.0.0.25 80 HTTP UDP 3100 +2010-12-01 23:45:54 Atlanta 10.0.0.1 55806 Texas 10.0.0.28 80 HTTP UDP 3009 +2010-12-01 23:45:55 Atlanta 10.0.0.1 58578 Texas 10.0.0.27 80 HTTP UDP 3419 +2010-12-01 23:45:56 Atlanta 10.0.0.1 50677 Texas 10.0.0.28 80 HTTP UDP 2924 +2010-12-01 23:45:57 Atlanta 10.0.0.1 50095 Texas 10.0.0.30 80 HTTP UDP 5414 +2010-12-01 23:45:58 Atlanta 10.0.0.1 56141 Texas 10.0.0.27 80 HTTP UDP 5039 +2010-12-01 23:45:59 Atlanta 10.0.0.1 65227 Texas 10.0.0.26 80 HTTP UDP 3996 +2010-12-01 23:46:00 Atlanta 10.0.0.1 62835 Texas 10.0.0.27 80 HTTP UDP 5484 +2010-12-01 23:46:01 Atlanta 10.0.0.1 56430 Texas 10.0.0.26 80 HTTP UDP 5092 +2010-12-01 23:46:02 Atlanta 10.0.0.1 51393 Texas 10.0.0.25 80 HTTP UDP 3857 +2010-12-01 23:46:03 Atlanta 10.0.0.1 55089 Texas 10.0.0.25 80 HTTP UDP 2603 +2010-12-01 23:46:04 Atlanta 10.0.0.1 52029 Texas 10.0.0.26 80 HTTP UDP 5275 +2010-12-01 23:46:05 Atlanta 10.0.0.1 50099 Texas 10.0.0.25 80 HTTP UDP 2813 +2010-12-01 23:46:06 Atlanta 10.0.0.1 62727 Texas 10.0.0.26 80 HTTP UDP 3223 +2010-12-01 23:46:07 Atlanta 10.0.0.1 50379 Texas 10.0.0.27 80 HTTP UDP 2956 +2010-12-01 23:46:08 Atlanta 10.0.0.1 58417 Texas 10.0.0.26 80 HTTP UDP 2757 +2010-12-01 23:46:09 Atlanta 10.0.0.1 58654 Texas 10.0.0.27 80 HTTP UDP 3585 +2010-12-01 23:46:10 Atlanta 10.0.0.1 64601 Texas 10.0.0.28 80 HTTP UDP 4657 +2010-12-01 23:46:11 Atlanta 10.0.0.1 51886 Texas 10.0.0.29 80 HTTP UDP 3211 +2010-12-01 23:46:12 Atlanta 10.0.0.1 64677 Texas 10.0.0.30 80 HTTP UDP 2537 +2010-12-01 23:46:13 Atlanta 10.0.0.1 54274 Texas 10.0.0.25 80 HTTP UDP 4414 +2010-12-01 23:46:14 Atlanta 10.0.0.1 61447 Texas 10.0.0.25 80 HTTP UDP 5347 +2010-12-01 23:46:15 Atlanta 10.0.0.1 55687 Texas 10.0.0.29 80 HTTP UDP 2001 +2010-12-01 23:46:16 Atlanta 10.0.0.1 51385 Texas 10.0.0.25 80 HTTP UDP 5552 +2010-12-01 23:46:17 Atlanta 10.0.0.1 64856 Texas 10.0.0.30 80 HTTP UDP 3685 +2010-12-01 23:46:18 Atlanta 10.0.0.1 60824 Texas 10.0.0.25 80 HTTP UDP 3398 +2010-12-01 23:46:19 Atlanta 10.0.0.1 57251 Texas 10.0.0.29 80 HTTP UDP 3790 +2010-12-01 23:46:20 Atlanta 10.0.0.1 53872 Texas 10.0.0.29 80 HTTP UDP 5346 +2010-12-01 23:46:21 Atlanta 10.0.0.1 62057 Texas 10.0.0.25 80 HTTP UDP 4867 +2010-12-01 23:46:22 Atlanta 10.0.0.1 50437 Texas 10.0.0.29 80 HTTP UDP 2088 +2010-12-01 23:46:23 Atlanta 10.0.0.1 64357 Texas 10.0.0.28 80 HTTP UDP 2956 +2010-12-01 23:46:24 Atlanta 10.0.0.1 55414 Texas 10.0.0.29 80 HTTP UDP 3471 +2010-12-01 23:46:25 Atlanta 10.0.0.1 55703 Texas 10.0.0.29 80 HTTP UDP 5628 +2010-12-01 23:46:26 Atlanta 10.0.0.1 51385 Texas 10.0.0.30 80 HTTP UDP 3634 +2010-12-01 23:46:27 Atlanta 10.0.0.1 53820 Texas 10.0.0.29 80 HTTP UDP 4794 +2010-12-01 23:46:28 Atlanta 10.0.0.1 60382 Texas 10.0.0.26 80 HTTP UDP 4711 +2010-12-01 23:46:29 Atlanta 10.0.0.1 49707 Texas 10.0.0.30 80 HTTP UDP 4868 +2010-12-01 23:46:30 Atlanta 10.0.0.1 59615 Texas 10.0.0.26 80 HTTP UDP 2133 +2010-12-01 23:46:31 Atlanta 10.0.0.1 59250 Texas 10.0.0.30 80 HTTP UDP 2595 +2010-12-01 23:46:32 Atlanta 10.0.0.1 62338 Texas 10.0.0.28 80 HTTP UDP 4935 +2010-12-01 23:46:33 Atlanta 10.0.0.1 55555 Texas 10.0.0.27 80 HTTP UDP 2784 +2010-12-01 23:46:34 Atlanta 10.0.0.1 55647 Texas 10.0.0.30 80 HTTP UDP 5593 +2010-12-01 23:46:35 Atlanta 10.0.0.1 56957 Texas 10.0.0.26 80 HTTP UDP 4780 +2010-12-01 23:46:36 Atlanta 10.0.0.1 64786 Texas 10.0.0.25 80 HTTP UDP 4646 +2010-12-01 23:46:37 Atlanta 10.0.0.1 59063 Texas 10.0.0.30 80 HTTP UDP 4289 +2010-12-01 23:46:38 Atlanta 10.0.0.1 61624 Texas 10.0.0.26 80 HTTP UDP 2819 +2010-12-01 23:46:39 Atlanta 10.0.0.1 57676 Texas 10.0.0.28 80 HTTP UDP 3969 +2010-12-01 23:46:40 Atlanta 10.0.0.1 60411 Texas 10.0.0.27 80 HTTP UDP 2569 +2010-12-01 23:46:41 Atlanta 10.0.0.1 55798 Texas 10.0.0.27 80 HTTP UDP 4647 +2010-12-01 23:46:42 Atlanta 10.0.0.1 54216 Texas 10.0.0.25 80 HTTP UDP 3034 +2010-12-01 23:46:43 Atlanta 10.0.0.1 59752 Texas 10.0.0.29 80 HTTP UDP 2994 +2010-12-01 23:46:44 Atlanta 10.0.0.1 49551 Texas 10.0.0.28 80 HTTP UDP 4043 +2010-12-01 23:46:45 Atlanta 10.0.0.1 54054 Texas 10.0.0.27 80 HTTP UDP 4363 +2010-12-01 23:46:46 Atlanta 10.0.0.1 49588 Texas 10.0.0.26 80 HTTP UDP 3565 +2010-12-01 23:46:47 Atlanta 10.0.0.1 61923 Texas 10.0.0.26 80 HTTP UDP 4041 +2010-12-01 23:46:48 Atlanta 10.0.0.1 53061 Texas 10.0.0.29 80 HTTP UDP 2975 +2010-12-01 23:46:49 Atlanta 10.0.0.1 55375 Texas 10.0.0.29 80 HTTP UDP 2857 +2010-12-01 23:46:50 Atlanta 10.0.0.1 59459 Texas 10.0.0.26 80 HTTP UDP 4041 +2010-12-01 23:46:51 Atlanta 10.0.0.1 51933 Texas 10.0.0.28 80 HTTP UDP 3146 +2010-12-01 23:46:52 Atlanta 10.0.0.1 54681 Texas 10.0.0.27 80 HTTP UDP 5593 +2010-12-01 23:46:53 Atlanta 10.0.0.1 54705 Texas 10.0.0.26 80 HTTP UDP 4669 +2010-12-01 23:46:54 Atlanta 10.0.0.1 61309 Texas 10.0.0.26 80 HTTP UDP 5671 +2010-12-01 23:46:55 Atlanta 10.0.0.1 54422 Texas 10.0.0.28 80 HTTP UDP 2554 +2010-12-01 23:46:56 Atlanta 10.0.0.1 51081 Texas 10.0.0.30 80 HTTP UDP 4827 +2010-12-01 23:46:57 Atlanta 10.0.0.1 59729 Texas 10.0.0.30 80 HTTP UDP 5850 +2010-12-01 23:46:58 Atlanta 10.0.0.1 50055 Texas 10.0.0.30 80 HTTP UDP 2133 +2010-12-01 23:46:59 Atlanta 10.0.0.1 50443 Texas 10.0.0.30 80 HTTP UDP 3398 +2010-12-01 23:47:00 Atlanta 10.0.0.1 57121 Texas 10.0.0.26 80 HTTP UDP 5435 +2010-12-01 23:47:01 Atlanta 10.0.0.1 52710 Texas 10.0.0.30 80 HTTP UDP 4195 +2010-12-01 23:47:02 Atlanta 10.0.0.1 56066 Texas 10.0.0.27 80 HTTP UDP 4032 +2010-12-01 23:47:03 Atlanta 10.0.0.1 49156 Texas 10.0.0.26 80 HTTP UDP 5759 +2010-12-01 23:47:04 Atlanta 10.0.0.1 56920 Texas 10.0.0.29 80 HTTP UDP 4935 +2010-12-01 23:47:05 Atlanta 10.0.0.1 50581 Texas 10.0.0.27 80 HTTP UDP 2114 +2010-12-01 23:47:06 Atlanta 10.0.0.1 57106 Texas 10.0.0.30 80 HTTP UDP 5321 +2010-12-01 23:47:07 Atlanta 10.0.0.1 53916 Texas 10.0.0.29 80 HTTP UDP 5041 +2010-12-01 23:47:08 Atlanta 10.0.0.1 63700 Texas 10.0.0.30 80 HTTP UDP 2746 +2010-12-01 23:47:09 Atlanta 10.0.0.1 55736 Texas 10.0.0.29 80 HTTP UDP 4369 +2010-12-01 23:47:10 Atlanta 10.0.0.1 56126 Texas 10.0.0.28 80 HTTP UDP 3318 +2010-12-01 23:47:11 Atlanta 10.0.0.1 58634 Texas 10.0.0.28 80 HTTP UDP 3107 +2010-12-01 23:47:12 Atlanta 10.0.0.1 60600 Texas 10.0.0.30 80 HTTP UDP 5482 +2010-12-01 23:47:13 Atlanta 10.0.0.1 51162 Texas 10.0.0.25 80 HTTP UDP 3951 +2010-12-01 23:47:14 Atlanta 10.0.0.1 51100 Texas 10.0.0.29 80 HTTP UDP 2322 +2010-12-01 23:47:15 Atlanta 10.0.0.1 56357 Texas 10.0.0.25 80 HTTP UDP 4102 +2010-12-01 23:47:16 Atlanta 10.0.0.1 55251 Texas 10.0.0.26 80 HTTP UDP 3571 +2010-12-01 23:47:17 Atlanta 10.0.0.1 56349 Texas 10.0.0.30 80 HTTP UDP 2231 +2010-12-01 23:47:18 Atlanta 10.0.0.1 60100 Texas 10.0.0.26 80 HTTP UDP 2921 +2010-12-01 23:47:19 Atlanta 10.0.0.1 58819 Texas 10.0.0.30 80 HTTP UDP 5217 +2010-12-01 23:47:20 Atlanta 10.0.0.1 56605 Texas 10.0.0.29 80 HTTP UDP 5706 +2010-12-01 23:47:21 Atlanta 10.0.0.1 58714 Texas 10.0.0.28 80 HTTP UDP 3562 +2010-12-01 23:47:22 Atlanta 10.0.0.1 62066 Texas 10.0.0.28 80 HTTP UDP 4744 +2010-12-01 23:47:23 Atlanta 10.0.0.1 49237 Texas 10.0.0.26 80 HTTP UDP 3764 +2010-12-01 23:47:24 Atlanta 10.0.0.1 55477 Texas 10.0.0.27 80 HTTP UDP 5860 +2010-12-01 23:47:25 Atlanta 10.0.0.1 54889 Texas 10.0.0.26 80 HTTP UDP 3203 +2010-12-01 23:47:26 Atlanta 10.0.0.1 63456 Texas 10.0.0.26 80 HTTP UDP 3786 +2010-12-01 23:47:27 Atlanta 10.0.0.1 50393 Texas 10.0.0.26 80 HTTP UDP 3896 +2010-12-01 23:47:28 Atlanta 10.0.0.1 49774 Texas 10.0.0.29 80 HTTP UDP 5341 +2010-12-01 23:47:29 Atlanta 10.0.0.1 52688 Texas 10.0.0.28 80 HTTP UDP 4173 +2010-12-01 23:47:30 Atlanta 10.0.0.1 61111 Texas 10.0.0.25 80 HTTP UDP 2366 +2010-12-01 23:47:31 Atlanta 10.0.0.1 64798 Texas 10.0.0.25 80 HTTP UDP 5667 +2010-12-01 23:47:32 Atlanta 10.0.0.1 58559 Texas 10.0.0.30 80 HTTP UDP 4179 +2010-12-01 23:47:33 Atlanta 10.0.0.1 53705 Texas 10.0.0.30 80 HTTP UDP 2189 +2010-12-01 23:47:34 Atlanta 10.0.0.1 59887 Texas 10.0.0.25 80 HTTP UDP 3786 +2010-12-01 23:47:35 Atlanta 10.0.0.1 50467 Texas 10.0.0.29 80 HTTP UDP 3114 +2010-12-01 23:47:36 Atlanta 10.0.0.1 56820 Texas 10.0.0.29 80 HTTP UDP 2242 +2010-12-01 23:47:37 Atlanta 10.0.0.1 54518 Texas 10.0.0.30 80 HTTP UDP 3450 +2010-12-01 23:47:38 Atlanta 10.0.0.1 63631 Texas 10.0.0.27 80 HTTP UDP 4046 +2010-12-01 23:47:39 Atlanta 10.0.0.1 51159 Texas 10.0.0.25 80 HTTP UDP 4038 +2010-12-01 23:47:40 Atlanta 10.0.0.1 50657 Texas 10.0.0.30 80 HTTP UDP 5243 +2010-12-01 23:47:41 Atlanta 10.0.0.1 62008 Texas 10.0.0.30 80 HTTP UDP 5850 +2010-12-01 23:47:42 Atlanta 10.0.0.1 60625 Texas 10.0.0.28 80 HTTP UDP 3597 +2010-12-01 23:47:43 Atlanta 10.0.0.1 57039 Texas 10.0.0.28 80 HTTP UDP 3851 +2010-12-01 23:47:44 Atlanta 10.0.0.1 50803 Texas 10.0.0.27 80 HTTP UDP 5673 +2010-12-01 23:47:45 Atlanta 10.0.0.1 62588 Texas 10.0.0.26 80 HTTP UDP 2788 +2010-12-01 23:47:46 Atlanta 10.0.0.1 64196 Texas 10.0.0.30 80 HTTP UDP 3761 +2010-12-01 23:47:47 Atlanta 10.0.0.1 50405 Texas 10.0.0.26 80 HTTP UDP 5810 +2010-12-01 23:47:48 Atlanta 10.0.0.1 58177 Texas 10.0.0.30 80 HTTP UDP 3433 +2010-12-01 23:47:49 Atlanta 10.0.0.1 54210 Texas 10.0.0.30 80 HTTP UDP 2029 +2010-12-01 23:47:50 Atlanta 10.0.0.1 55028 Texas 10.0.0.25 80 HTTP UDP 5963 +2010-12-01 23:47:51 Atlanta 10.0.0.1 63512 Texas 10.0.0.27 80 HTTP UDP 4826 +2010-12-01 23:47:52 Atlanta 10.0.0.1 57775 Texas 10.0.0.30 80 HTTP UDP 2024 +2010-12-01 23:47:53 Atlanta 10.0.0.1 53766 Texas 10.0.0.25 80 HTTP UDP 2640 +2010-12-01 23:47:54 Atlanta 10.0.0.1 58989 Texas 10.0.0.29 80 HTTP UDP 5004 +2010-12-01 23:47:55 Atlanta 10.0.0.1 53315 Texas 10.0.0.27 80 HTTP UDP 4635 +2010-12-01 23:47:56 Atlanta 10.0.0.1 50498 Texas 10.0.0.30 80 HTTP UDP 3224 +2010-12-01 23:47:57 Atlanta 10.0.0.1 64429 Texas 10.0.0.29 80 HTTP UDP 5668 +2010-12-01 23:47:58 Atlanta 10.0.0.1 50878 Texas 10.0.0.27 80 HTTP UDP 5835 +2010-12-01 23:47:59 Atlanta 10.0.0.1 65014 Texas 10.0.0.25 80 HTTP UDP 2447 +2010-12-01 23:48:00 Atlanta 10.0.0.1 61456 Texas 10.0.0.29 80 HTTP UDP 2294 +2010-12-01 23:48:01 Atlanta 10.0.0.1 63939 Texas 10.0.0.25 80 HTTP UDP 2038 +2010-12-01 23:48:02 Atlanta 10.0.0.1 64670 Texas 10.0.0.30 80 HTTP UDP 5082 +2010-12-01 23:48:03 Atlanta 10.0.0.1 58338 Texas 10.0.0.27 80 HTTP UDP 4165 +2010-12-01 23:48:04 Atlanta 10.0.0.1 59341 Texas 10.0.0.28 80 HTTP UDP 3646 +2010-12-01 23:48:05 Atlanta 10.0.0.1 65261 Texas 10.0.0.26 80 HTTP UDP 4955 +2010-12-01 23:48:06 Atlanta 10.0.0.1 56860 Texas 10.0.0.27 80 HTTP UDP 2311 +2010-12-01 23:48:07 Atlanta 10.0.0.1 58676 Texas 10.0.0.30 80 HTTP UDP 4369 +2010-12-01 23:48:08 Atlanta 10.0.0.1 60614 Texas 10.0.0.30 80 HTTP UDP 3293 +2010-12-01 23:48:09 Atlanta 10.0.0.1 61714 Texas 10.0.0.29 80 HTTP UDP 4279 +2010-12-01 23:48:10 Atlanta 10.0.0.1 50720 Texas 10.0.0.27 80 HTTP UDP 2684 +2010-12-01 23:48:11 Atlanta 10.0.0.1 49375 Texas 10.0.0.26 80 HTTP UDP 5506 +2010-12-01 23:48:12 Atlanta 10.0.0.1 51719 Texas 10.0.0.28 80 HTTP UDP 2656 +2010-12-01 23:48:13 Atlanta 10.0.0.1 53014 Texas 10.0.0.28 80 HTTP UDP 5191 +2010-12-01 23:48:14 Atlanta 10.0.0.1 51082 Texas 10.0.0.30 80 HTTP UDP 3731 +2010-12-01 23:48:15 Atlanta 10.0.0.1 60117 Texas 10.0.0.30 80 HTTP UDP 4771 +2010-12-01 23:48:16 Atlanta 10.0.0.1 59910 Texas 10.0.0.27 80 HTTP UDP 2940 +2010-12-01 23:48:17 Atlanta 10.0.0.1 58357 Texas 10.0.0.28 80 HTTP UDP 3318 +2010-12-01 23:48:18 Atlanta 10.0.0.1 49605 Texas 10.0.0.25 80 HTTP UDP 2548 +2010-12-01 23:48:19 Atlanta 10.0.0.1 50130 Texas 10.0.0.28 80 HTTP UDP 2643 +2010-12-01 23:48:20 Atlanta 10.0.0.1 51953 Texas 10.0.0.26 80 HTTP UDP 4826 +2010-12-01 23:48:21 Atlanta 10.0.0.1 58918 Texas 10.0.0.28 80 HTTP UDP 3241 +2010-12-01 23:48:22 Atlanta 10.0.0.1 63986 Texas 10.0.0.27 80 HTTP UDP 2462 +2010-12-01 23:48:23 Atlanta 10.0.0.1 57230 Texas 10.0.0.28 80 HTTP UDP 4600 +2010-12-01 23:48:24 Atlanta 10.0.0.1 54078 Texas 10.0.0.25 80 HTTP UDP 4040 +2010-12-01 23:48:25 Atlanta 10.0.0.1 53625 Texas 10.0.0.30 80 HTTP UDP 4377 +2010-12-01 23:48:26 Atlanta 10.0.0.1 54292 Texas 10.0.0.30 80 HTTP UDP 3009 +2010-12-01 23:48:27 Atlanta 10.0.0.1 49936 Texas 10.0.0.29 80 HTTP UDP 5019 +2010-12-01 23:48:28 Atlanta 10.0.0.1 51071 Texas 10.0.0.25 80 HTTP UDP 4549 +2010-12-01 23:48:29 Atlanta 10.0.0.1 53205 Texas 10.0.0.28 80 HTTP UDP 2855 +2010-12-01 23:48:30 Atlanta 10.0.0.1 50688 Texas 10.0.0.27 80 HTTP UDP 4683 +2010-12-01 23:48:31 Atlanta 10.0.0.1 56832 Texas 10.0.0.30 80 HTTP UDP 5911 +2010-12-01 23:48:32 Atlanta 10.0.0.1 61511 Texas 10.0.0.28 80 HTTP UDP 5171 +2010-12-01 23:48:33 Atlanta 10.0.0.1 60915 Texas 10.0.0.28 80 HTTP UDP 3226 +2010-12-01 23:48:34 Atlanta 10.0.0.1 51194 Texas 10.0.0.27 80 HTTP UDP 2054 +2010-12-01 23:48:35 Atlanta 10.0.0.1 50000 Texas 10.0.0.27 80 HTTP UDP 5946 +2010-12-01 23:48:36 Atlanta 10.0.0.1 54636 Texas 10.0.0.30 80 HTTP UDP 3070 +2010-12-01 23:48:37 Atlanta 10.0.0.1 51676 Texas 10.0.0.29 80 HTTP UDP 2003 +2010-12-01 23:48:38 Atlanta 10.0.0.1 61808 Texas 10.0.0.30 80 HTTP UDP 3445 +2010-12-01 23:48:39 Atlanta 10.0.0.1 64864 Texas 10.0.0.29 80 HTTP UDP 3416 +2010-12-01 23:48:40 Atlanta 10.0.0.1 58217 Texas 10.0.0.27 80 HTTP UDP 5480 +2010-12-01 23:48:41 Atlanta 10.0.0.1 57616 Texas 10.0.0.29 80 HTTP UDP 4735 +2010-12-01 23:48:42 Atlanta 10.0.0.1 58277 Texas 10.0.0.26 80 HTTP UDP 3171 +2010-12-01 23:48:43 Atlanta 10.0.0.1 63294 Texas 10.0.0.25 80 HTTP UDP 4918 +2010-12-01 23:48:44 Atlanta 10.0.0.1 62937 Texas 10.0.0.29 80 HTTP UDP 5369 +2010-12-01 23:48:45 Atlanta 10.0.0.1 62225 Texas 10.0.0.29 80 HTTP UDP 5321 +2010-12-01 23:48:46 Atlanta 10.0.0.1 52696 Texas 10.0.0.26 80 HTTP UDP 3476 +2010-12-01 23:48:47 Atlanta 10.0.0.1 52869 Texas 10.0.0.25 80 HTTP UDP 3066 +2010-12-01 23:48:48 Atlanta 10.0.0.1 56312 Texas 10.0.0.25 80 HTTP UDP 3376 +2010-12-01 23:48:49 Atlanta 10.0.0.1 49227 Texas 10.0.0.26 80 HTTP UDP 3621 +2010-12-01 23:48:50 Atlanta 10.0.0.1 60784 Texas 10.0.0.27 80 HTTP UDP 4515 +2010-12-01 23:48:51 Atlanta 10.0.0.1 50890 Texas 10.0.0.27 80 HTTP UDP 2577 +2010-12-01 23:48:52 Atlanta 10.0.0.1 56433 Texas 10.0.0.30 80 HTTP UDP 4324 +2010-12-01 23:48:53 Atlanta 10.0.0.1 50873 Texas 10.0.0.28 80 HTTP UDP 2818 +2010-12-01 23:48:54 Atlanta 10.0.0.1 52613 Texas 10.0.0.30 80 HTTP UDP 4627 +2010-12-01 23:48:55 Atlanta 10.0.0.1 64951 Texas 10.0.0.27 80 HTTP UDP 3171 +2010-12-01 23:48:56 Atlanta 10.0.0.1 56667 Texas 10.0.0.25 80 HTTP UDP 2003 +2010-12-01 23:48:57 Atlanta 10.0.0.1 52833 Texas 10.0.0.29 80 HTTP UDP 2462 +2010-12-01 23:48:58 Atlanta 10.0.0.1 62710 Texas 10.0.0.27 80 HTTP UDP 4138 +2010-12-01 23:48:59 Atlanta 10.0.0.1 58306 Texas 10.0.0.27 80 HTTP UDP 2577 +2010-12-01 23:49:00 Atlanta 10.0.0.1 64081 Texas 10.0.0.30 80 HTTP UDP 2094 +2010-12-01 23:49:01 Atlanta 10.0.0.1 60582 Texas 10.0.0.26 80 HTTP UDP 2366 +2010-12-01 23:49:02 Atlanta 10.0.0.1 62083 Texas 10.0.0.26 80 HTTP UDP 2640 +2010-12-01 23:49:03 Atlanta 10.0.0.1 51109 Texas 10.0.0.25 80 HTTP UDP 5705 +2010-12-01 23:49:04 Atlanta 10.0.0.1 60600 Texas 10.0.0.25 80 HTTP UDP 2480 +2010-12-01 23:49:05 Atlanta 10.0.0.1 51700 Texas 10.0.0.26 80 HTTP UDP 2996 +2010-12-01 23:49:06 Atlanta 10.0.0.1 61886 Texas 10.0.0.30 80 HTTP UDP 2298 +2010-12-01 23:49:07 Atlanta 10.0.0.1 56025 Texas 10.0.0.26 80 HTTP UDP 3216 +2010-12-01 23:49:08 Atlanta 10.0.0.1 65437 Texas 10.0.0.29 80 HTTP UDP 3970 +2010-12-01 23:49:09 Atlanta 10.0.0.1 54686 Texas 10.0.0.25 80 HTTP UDP 4090 +2010-12-01 23:49:10 Atlanta 10.0.0.1 63992 Texas 10.0.0.29 80 HTTP UDP 5338 +2010-12-01 23:49:11 Atlanta 10.0.0.1 55516 Texas 10.0.0.25 80 HTTP UDP 5868 +2010-12-01 23:49:12 Atlanta 10.0.0.1 57641 Texas 10.0.0.29 80 HTTP UDP 5267 +2010-12-01 23:49:13 Atlanta 10.0.0.1 57408 Texas 10.0.0.30 80 HTTP UDP 4154 +2010-12-01 23:49:14 Atlanta 10.0.0.1 63080 Texas 10.0.0.30 80 HTTP UDP 5309 +2010-12-01 23:49:15 Atlanta 10.0.0.1 49916 Texas 10.0.0.27 80 HTTP UDP 4449 +2010-12-01 23:49:16 Atlanta 10.0.0.1 65331 Texas 10.0.0.27 80 HTTP UDP 5371 +2010-12-01 23:49:17 Atlanta 10.0.0.1 61702 Texas 10.0.0.26 80 HTTP UDP 3970 +2010-12-01 23:49:18 Atlanta 10.0.0.1 50140 Texas 10.0.0.29 80 HTTP UDP 5169 +2010-12-01 23:49:19 Atlanta 10.0.0.1 58015 Texas 10.0.0.27 80 HTTP UDP 4414 +2010-12-01 23:49:20 Atlanta 10.0.0.1 61609 Texas 10.0.0.25 80 HTTP UDP 2144 +2010-12-01 23:49:21 Atlanta 10.0.0.1 52622 Texas 10.0.0.28 80 HTTP UDP 2231 +2010-12-01 23:49:22 Atlanta 10.0.0.1 50948 Texas 10.0.0.28 80 HTTP UDP 5749 +2010-12-01 23:49:23 Atlanta 10.0.0.1 65457 Texas 10.0.0.25 80 HTTP UDP 2192 +2010-12-01 23:49:24 Atlanta 10.0.0.1 53716 Texas 10.0.0.27 80 HTTP UDP 3565 +2010-12-01 23:49:25 Atlanta 10.0.0.1 61746 Texas 10.0.0.30 80 HTTP UDP 4886 +2010-12-01 23:49:26 Atlanta 10.0.0.1 53815 Texas 10.0.0.25 80 HTTP UDP 4013 +2010-12-01 23:49:27 Atlanta 10.0.0.1 63110 Texas 10.0.0.30 80 HTTP UDP 4359 +2010-12-01 23:49:28 Atlanta 10.0.0.1 54635 Texas 10.0.0.26 80 HTTP UDP 5380 +2010-12-01 23:49:29 Atlanta 10.0.0.1 53900 Texas 10.0.0.28 80 HTTP UDP 3067 +2010-12-01 23:49:30 Atlanta 10.0.0.1 56607 Texas 10.0.0.25 80 HTTP UDP 5810 +2010-12-01 23:49:31 Atlanta 10.0.0.1 51682 Texas 10.0.0.25 80 HTTP UDP 4969 +2010-12-01 23:49:32 Atlanta 10.0.0.1 62984 Texas 10.0.0.27 80 HTTP UDP 4171 +2010-12-01 23:49:33 Atlanta 10.0.0.1 65375 Texas 10.0.0.28 80 HTTP UDP 5705 +2010-12-01 23:49:34 Atlanta 10.0.0.1 49730 Texas 10.0.0.29 80 HTTP UDP 4948 +2010-12-01 23:49:35 Atlanta 10.0.0.1 56716 Texas 10.0.0.30 80 HTTP UDP 5223 +2010-12-01 23:49:36 Atlanta 10.0.0.1 60956 Texas 10.0.0.28 80 HTTP UDP 3285 +2010-12-01 23:49:37 Atlanta 10.0.0.1 57142 Texas 10.0.0.30 80 HTTP UDP 4154 +2010-12-01 23:49:38 Atlanta 10.0.0.1 60576 Texas 10.0.0.27 80 HTTP UDP 4893 +2010-12-01 23:49:39 Atlanta 10.0.0.1 57701 Texas 10.0.0.30 80 HTTP UDP 5163 +2010-12-01 23:49:40 Atlanta 10.0.0.1 59909 Texas 10.0.0.30 80 HTTP UDP 5952 +2010-12-01 23:49:41 Atlanta 10.0.0.1 57459 Texas 10.0.0.26 80 HTTP UDP 5326 +2010-12-01 23:49:42 Atlanta 10.0.0.1 56659 Texas 10.0.0.29 80 HTTP UDP 4647 +2010-12-01 23:49:43 Atlanta 10.0.0.1 59189 Texas 10.0.0.30 80 HTTP UDP 2561 +2010-12-01 23:49:44 Atlanta 10.0.0.1 55529 Texas 10.0.0.26 80 HTTP UDP 2561 +2010-12-01 23:49:45 Atlanta 10.0.0.1 55273 Texas 10.0.0.29 80 HTTP UDP 5667 +2010-12-01 23:49:46 Atlanta 10.0.0.1 56570 Texas 10.0.0.29 80 HTTP UDP 2617 +2010-12-01 23:49:47 Atlanta 10.0.0.1 61576 Texas 10.0.0.25 80 HTTP UDP 3402 +2010-12-01 23:49:48 Atlanta 10.0.0.1 53918 Texas 10.0.0.30 80 HTTP UDP 3228 +2010-12-01 23:49:49 Atlanta 10.0.0.1 62337 Texas 10.0.0.25 80 HTTP UDP 4546 +2010-12-01 23:49:50 Atlanta 10.0.0.1 60427 Texas 10.0.0.30 80 HTTP UDP 3511 +2010-12-01 23:49:51 Atlanta 10.0.0.1 49690 Texas 10.0.0.25 80 HTTP UDP 4042 +2010-12-01 23:49:52 Atlanta 10.0.0.1 50110 Texas 10.0.0.30 80 HTTP UDP 2805 +2010-12-01 23:49:53 Atlanta 10.0.0.1 60971 Texas 10.0.0.25 80 HTTP UDP 5946 +2010-12-01 23:49:54 Atlanta 10.0.0.1 61244 Texas 10.0.0.30 80 HTTP UDP 4297 +2010-12-01 23:49:55 Atlanta 10.0.0.1 51743 Texas 10.0.0.28 80 HTTP UDP 2611 +2010-12-01 23:49:56 Atlanta 10.0.0.1 57811 Texas 10.0.0.30 80 HTTP UDP 3521 +2010-12-01 23:49:57 Atlanta 10.0.0.1 62192 Texas 10.0.0.30 80 HTTP UDP 5925 +2010-12-01 23:49:58 Atlanta 10.0.0.1 51815 Texas 10.0.0.30 80 HTTP UDP 2782 +2010-12-01 23:49:59 Atlanta 10.0.0.1 55004 Texas 10.0.0.27 80 HTTP UDP 4258 +2010-12-01 23:50:00 Atlanta 10.0.0.1 55661 Texas 10.0.0.29 80 HTTP UDP 3788 +2010-12-01 23:50:01 Atlanta 10.0.0.1 54831 Texas 10.0.0.28 80 HTTP UDP 5541 +2010-12-01 23:50:02 Atlanta 10.0.0.1 53145 Texas 10.0.0.30 80 HTTP UDP 4349 +2010-12-01 23:50:03 Atlanta 10.0.0.1 51156 Texas 10.0.0.25 80 HTTP UDP 3433 +2010-12-01 23:50:04 Atlanta 10.0.0.1 62121 Texas 10.0.0.28 80 HTTP UDP 3955 +2010-12-01 23:50:05 Atlanta 10.0.0.1 58742 Texas 10.0.0.25 80 HTTP UDP 5948 +2010-12-01 23:50:06 Atlanta 10.0.0.1 52237 Texas 10.0.0.26 80 HTTP UDP 4925 +2010-12-01 23:50:07 Atlanta 10.0.0.1 57640 Texas 10.0.0.27 80 HTTP UDP 4258 +2010-12-01 23:50:08 Atlanta 10.0.0.1 60507 Texas 10.0.0.29 80 HTTP UDP 2850 +2010-12-01 23:50:09 Atlanta 10.0.0.1 49223 Texas 10.0.0.25 80 HTTP UDP 5807 +2010-12-01 23:50:10 Atlanta 10.0.0.1 51935 Texas 10.0.0.25 80 HTTP UDP 3834 +2010-12-01 23:50:11 Atlanta 10.0.0.1 54177 Texas 10.0.0.27 80 HTTP UDP 5346 +2010-12-01 23:50:12 Atlanta 10.0.0.1 61877 Texas 10.0.0.28 80 HTTP UDP 5231 +2010-12-01 23:50:13 Atlanta 10.0.0.1 64457 Texas 10.0.0.28 80 HTTP UDP 3872 +2010-12-01 23:50:14 Atlanta 10.0.0.1 63618 Texas 10.0.0.29 80 HTTP UDP 3549 +2010-12-01 23:50:15 Atlanta 10.0.0.1 57469 Texas 10.0.0.29 80 HTTP UDP 5570 +2010-12-01 23:50:16 Atlanta 10.0.0.1 53807 Texas 10.0.0.25 80 HTTP UDP 3900 +2010-12-01 23:50:17 Atlanta 10.0.0.1 64699 Texas 10.0.0.30 80 HTTP UDP 3198 +2010-12-01 23:50:18 Atlanta 10.0.0.1 58000 Texas 10.0.0.27 80 HTTP UDP 5747 +2010-12-01 23:50:19 Atlanta 10.0.0.1 51324 Texas 10.0.0.29 80 HTTP UDP 4751 +2010-12-01 23:50:20 Atlanta 10.0.0.1 55766 Texas 10.0.0.28 80 HTTP UDP 2688 +2010-12-01 23:50:21 Atlanta 10.0.0.1 52861 Texas 10.0.0.25 80 HTTP UDP 4238 +2010-12-01 23:50:22 Atlanta 10.0.0.1 56728 Texas 10.0.0.25 80 HTTP UDP 4696 +2010-12-01 23:50:23 Atlanta 10.0.0.1 53457 Texas 10.0.0.30 80 HTTP UDP 4358 +2010-12-01 23:50:24 Atlanta 10.0.0.1 50650 Texas 10.0.0.25 80 HTTP UDP 2295 +2010-12-01 23:50:25 Atlanta 10.0.0.1 61474 Texas 10.0.0.29 80 HTTP UDP 3958 +2010-12-01 23:50:26 Atlanta 10.0.0.1 56555 Texas 10.0.0.26 80 HTTP UDP 2523 +2010-12-01 23:50:27 Atlanta 10.0.0.1 57051 Texas 10.0.0.30 80 HTTP UDP 5856 +2010-12-01 23:50:28 Atlanta 10.0.0.1 61576 Texas 10.0.0.26 80 HTTP UDP 4753 +2010-12-01 23:50:29 Atlanta 10.0.0.1 60331 Texas 10.0.0.30 80 HTTP UDP 3114 +2010-12-01 23:50:30 Atlanta 10.0.0.1 60621 Texas 10.0.0.27 80 HTTP UDP 4290 +2010-12-01 23:50:31 Atlanta 10.0.0.1 56749 Texas 10.0.0.29 80 HTTP UDP 2891 +2010-12-01 23:50:32 Atlanta 10.0.0.1 55947 Texas 10.0.0.29 80 HTTP UDP 4501 +2010-12-01 23:50:33 Atlanta 10.0.0.1 59467 Texas 10.0.0.30 80 HTTP UDP 3466 +2010-12-01 23:50:34 Atlanta 10.0.0.1 57464 Texas 10.0.0.30 80 HTTP UDP 3609 +2010-12-01 23:50:35 Atlanta 10.0.0.1 65355 Texas 10.0.0.28 80 HTTP UDP 2690 +2010-12-01 23:50:36 Atlanta 10.0.0.1 59007 Texas 10.0.0.30 80 HTTP UDP 5584 +2010-12-01 23:50:37 Atlanta 10.0.0.1 52575 Texas 10.0.0.28 80 HTTP UDP 5937 +2010-12-01 23:50:38 Atlanta 10.0.0.1 52820 Texas 10.0.0.30 80 HTTP UDP 4411 +2010-12-01 23:50:39 Atlanta 10.0.0.1 63585 Texas 10.0.0.26 80 HTTP UDP 3501 +2010-12-01 23:50:40 Atlanta 10.0.0.1 62761 Texas 10.0.0.29 80 HTTP UDP 2432 +2010-12-01 23:50:41 Atlanta 10.0.0.1 49736 Texas 10.0.0.30 80 HTTP UDP 3318 +2010-12-01 23:50:42 Atlanta 10.0.0.1 51915 Texas 10.0.0.25 80 HTTP UDP 2029 +2010-12-01 23:50:43 Atlanta 10.0.0.1 62279 Texas 10.0.0.29 80 HTTP UDP 3658 +2010-12-01 23:50:44 Atlanta 10.0.0.1 53232 Texas 10.0.0.27 80 HTTP UDP 2595 +2010-12-01 23:50:45 Atlanta 10.0.0.1 54825 Texas 10.0.0.30 80 HTTP UDP 2532 +2010-12-01 23:50:46 Atlanta 10.0.0.1 53402 Texas 10.0.0.28 80 HTTP UDP 3521 +2010-12-01 23:50:47 Atlanta 10.0.0.1 52905 Texas 10.0.0.27 80 HTTP UDP 3223 +2010-12-01 23:50:48 Atlanta 10.0.0.1 62888 Texas 10.0.0.29 80 HTTP UDP 5542 +2010-12-01 23:50:49 Atlanta 10.0.0.1 59115 Texas 10.0.0.27 80 HTTP UDP 5171 +2010-12-01 23:50:50 Atlanta 10.0.0.1 52144 Texas 10.0.0.29 80 HTTP UDP 2144 +2010-12-01 23:50:51 Atlanta 10.0.0.1 52315 Texas 10.0.0.25 80 HTTP UDP 5348 +2010-12-01 23:50:52 Atlanta 10.0.0.1 64265 Texas 10.0.0.25 80 HTTP UDP 2598 +2010-12-01 23:50:53 Atlanta 10.0.0.1 58810 Texas 10.0.0.30 80 HTTP UDP 2602 +2010-12-01 23:50:54 Atlanta 10.0.0.1 62658 Texas 10.0.0.30 80 HTTP UDP 2168 +2010-12-01 23:50:55 Atlanta 10.0.0.1 56050 Texas 10.0.0.26 80 HTTP UDP 5309 +2010-12-01 23:50:56 Atlanta 10.0.0.1 60152 Texas 10.0.0.26 80 HTTP UDP 4899 +2010-12-01 23:50:57 Atlanta 10.0.0.1 54464 Texas 10.0.0.26 80 HTTP UDP 3893 +2010-12-01 23:50:58 Atlanta 10.0.0.1 56336 Texas 10.0.0.26 80 HTTP UDP 2479 +2010-12-01 23:50:59 Atlanta 10.0.0.1 64112 Texas 10.0.0.27 80 HTTP UDP 4524 +2010-12-01 23:51:00 Atlanta 10.0.0.1 63833 Texas 10.0.0.25 80 HTTP UDP 4058 +2010-12-01 23:51:01 Atlanta 10.0.0.1 51416 Texas 10.0.0.30 80 HTTP UDP 4753 +2010-12-01 23:51:02 Atlanta 10.0.0.1 59262 Texas 10.0.0.26 80 HTTP UDP 4359 +2010-12-01 23:51:03 Atlanta 10.0.0.1 57086 Texas 10.0.0.26 80 HTTP UDP 5232 +2010-12-01 23:51:04 Atlanta 10.0.0.1 49580 Texas 10.0.0.25 80 HTTP UDP 4804 +2010-12-01 23:51:05 Atlanta 10.0.0.1 56659 Texas 10.0.0.27 80 HTTP UDP 3630 +2010-12-01 23:51:06 Atlanta 10.0.0.1 56175 Texas 10.0.0.25 80 HTTP UDP 2054 +2010-12-01 23:51:07 Atlanta 10.0.0.1 54400 Texas 10.0.0.30 80 HTTP UDP 3958 +2010-12-01 23:51:08 Atlanta 10.0.0.1 53102 Texas 10.0.0.28 80 HTTP UDP 4751 +2010-12-01 23:51:09 Atlanta 10.0.0.1 60106 Texas 10.0.0.25 80 HTTP UDP 5998 +2010-12-01 23:51:10 Atlanta 10.0.0.1 60307 Texas 10.0.0.30 80 HTTP UDP 4189 +2010-12-01 23:51:11 Atlanta 10.0.0.1 58347 Texas 10.0.0.25 80 HTTP UDP 5125 +2010-12-01 23:51:12 Atlanta 10.0.0.1 60465 Texas 10.0.0.25 80 HTTP UDP 5135 +2010-12-01 23:51:13 Atlanta 10.0.0.1 54213 Texas 10.0.0.29 80 HTTP UDP 2963 +2010-12-01 23:51:14 Atlanta 10.0.0.1 52417 Texas 10.0.0.27 80 HTTP UDP 2954 +2010-12-01 23:51:15 Atlanta 10.0.0.1 57051 Texas 10.0.0.30 80 HTTP UDP 4088 +2010-12-01 23:51:16 Atlanta 10.0.0.1 58665 Texas 10.0.0.26 80 HTTP UDP 3935 +2010-12-01 23:51:17 Atlanta 10.0.0.1 54271 Texas 10.0.0.26 80 HTTP UDP 3189 +2010-12-01 23:51:18 Atlanta 10.0.0.1 53357 Texas 10.0.0.26 80 HTTP UDP 4566 +2010-12-01 23:51:19 Atlanta 10.0.0.1 59449 Texas 10.0.0.29 80 HTTP UDP 5835 +2010-12-01 23:51:20 Atlanta 10.0.0.1 61500 Texas 10.0.0.26 80 HTTP UDP 5371 +2010-12-01 23:51:21 Atlanta 10.0.0.1 49771 Texas 10.0.0.26 80 HTTP UDP 5047 +2010-12-01 23:51:22 Atlanta 10.0.0.1 49327 Texas 10.0.0.27 80 HTTP UDP 2178 +2010-12-01 23:51:23 Atlanta 10.0.0.1 56221 Texas 10.0.0.26 80 HTTP UDP 2994 +2010-12-01 23:51:24 Atlanta 10.0.0.1 62663 Texas 10.0.0.29 80 HTTP UDP 4735 +2010-12-01 23:51:25 Atlanta 10.0.0.1 53908 Texas 10.0.0.29 80 HTTP UDP 3104 +2010-12-01 23:51:26 Atlanta 10.0.0.1 53117 Texas 10.0.0.29 80 HTTP UDP 4488 +2010-12-01 23:51:27 Atlanta 10.0.0.1 64310 Texas 10.0.0.29 80 HTTP UDP 3908 +2010-12-01 23:51:28 Atlanta 10.0.0.1 49151 Texas 10.0.0.25 80 HTTP UDP 2656 +2010-12-01 23:51:29 Atlanta 10.0.0.1 62706 Texas 10.0.0.29 80 HTTP UDP 3523 +2010-12-01 23:51:30 Atlanta 10.0.0.1 55294 Texas 10.0.0.27 80 HTTP UDP 5309 +2010-12-01 23:51:31 Atlanta 10.0.0.1 63421 Texas 10.0.0.30 80 HTTP UDP 3182 +2010-12-01 23:51:32 Atlanta 10.0.0.1 53974 Texas 10.0.0.26 80 HTTP UDP 4290 +2010-12-01 23:51:33 Atlanta 10.0.0.1 61859 Texas 10.0.0.28 80 HTTP UDP 3845 +2010-12-01 23:51:34 Atlanta 10.0.0.1 63885 Texas 10.0.0.26 80 HTTP UDP 4922 +2010-12-01 23:51:35 Atlanta 10.0.0.1 54991 Texas 10.0.0.27 80 HTTP UDP 2047 +2010-12-01 23:51:36 Atlanta 10.0.0.1 64570 Texas 10.0.0.27 80 HTTP UDP 5940 +2010-12-01 23:51:37 Atlanta 10.0.0.1 52414 Texas 10.0.0.28 80 HTTP UDP 4864 +2010-12-01 23:51:38 Atlanta 10.0.0.1 51490 Texas 10.0.0.29 80 HTTP UDP 4711 +2010-12-01 23:51:39 Atlanta 10.0.0.1 63557 Texas 10.0.0.30 80 HTTP UDP 4549 +2010-12-01 23:51:40 Atlanta 10.0.0.1 63791 Texas 10.0.0.27 80 HTTP UDP 2997 +2010-12-01 23:51:41 Atlanta 10.0.0.1 53907 Texas 10.0.0.27 80 HTTP UDP 5952 +2010-12-01 23:51:42 Atlanta 10.0.0.1 53669 Texas 10.0.0.28 80 HTTP UDP 5252 +2010-12-01 23:51:43 Atlanta 10.0.0.1 54489 Texas 10.0.0.29 80 HTTP UDP 2611 +2010-12-01 23:51:44 Atlanta 10.0.0.1 62706 Texas 10.0.0.25 80 HTTP UDP 5849 +2010-12-01 23:51:45 Atlanta 10.0.0.1 51592 Texas 10.0.0.30 80 HTTP UDP 5545 +2010-12-01 23:51:46 Atlanta 10.0.0.1 55282 Texas 10.0.0.27 80 HTTP UDP 3471 +2010-12-01 23:51:47 Atlanta 10.0.0.1 50046 Texas 10.0.0.26 80 HTTP UDP 4750 +2010-12-01 23:51:48 Atlanta 10.0.0.1 56992 Texas 10.0.0.28 80 HTTP UDP 2945 +2010-12-01 23:51:49 Atlanta 10.0.0.1 56259 Texas 10.0.0.27 80 HTTP UDP 4506 +2010-12-01 23:51:50 Atlanta 10.0.0.1 60166 Texas 10.0.0.26 80 HTTP UDP 4287 +2010-12-01 23:51:51 Atlanta 10.0.0.1 65033 Texas 10.0.0.27 80 HTTP UDP 2643 +2010-12-01 23:51:52 Atlanta 10.0.0.1 65045 Texas 10.0.0.28 80 HTTP UDP 5501 +2010-12-01 23:51:53 Atlanta 10.0.0.1 50490 Texas 10.0.0.25 80 HTTP UDP 2237 +2010-12-01 23:51:54 Atlanta 10.0.0.1 55399 Texas 10.0.0.25 80 HTTP UDP 2906 +2010-12-01 23:51:55 Atlanta 10.0.0.1 56967 Texas 10.0.0.25 80 HTTP UDP 4587 +2010-12-01 23:51:56 Atlanta 10.0.0.1 52033 Texas 10.0.0.26 80 HTTP UDP 2240 +2010-12-01 23:51:57 Atlanta 10.0.0.1 53203 Texas 10.0.0.25 80 HTTP UDP 3501 +2010-12-01 23:51:58 Atlanta 10.0.0.1 54075 Texas 10.0.0.26 80 HTTP UDP 2578 +2010-12-01 23:51:59 Atlanta 10.0.0.1 56444 Texas 10.0.0.27 80 HTTP UDP 2954 +2010-12-01 23:52:00 Atlanta 10.0.0.1 55656 Texas 10.0.0.29 80 HTTP UDP 2545 +2010-12-01 23:52:01 Atlanta 10.0.0.1 54247 Texas 10.0.0.26 80 HTTP UDP 2590 +2010-12-01 23:52:02 Atlanta 10.0.0.1 51840 Texas 10.0.0.26 80 HTTP UDP 5255 +2010-12-01 23:52:03 Atlanta 10.0.0.1 51939 Texas 10.0.0.30 80 HTTP UDP 3121 +2010-12-01 23:52:04 Atlanta 10.0.0.1 55138 Texas 10.0.0.29 80 HTTP UDP 3028 +2010-12-01 23:52:05 Atlanta 10.0.0.1 59773 Texas 10.0.0.26 80 HTTP UDP 3918 +2010-12-01 23:52:06 Atlanta 10.0.0.1 63511 Texas 10.0.0.25 80 HTTP UDP 4751 +2010-12-01 23:52:07 Atlanta 10.0.0.1 62486 Texas 10.0.0.26 80 HTTP UDP 4644 +2010-12-01 23:52:08 Atlanta 10.0.0.1 64300 Texas 10.0.0.25 80 HTTP UDP 3104 +2010-12-01 23:52:09 Atlanta 10.0.0.1 63274 Texas 10.0.0.29 80 HTTP UDP 2313 +2010-12-01 23:52:10 Atlanta 10.0.0.1 52365 Texas 10.0.0.25 80 HTTP UDP 3374 +2010-12-01 23:52:11 Atlanta 10.0.0.1 59172 Texas 10.0.0.25 80 HTTP UDP 2480 +2010-12-01 23:52:12 Atlanta 10.0.0.1 53783 Texas 10.0.0.28 80 HTTP UDP 3732 +2010-12-01 23:52:13 Atlanta 10.0.0.1 58950 Texas 10.0.0.25 80 HTTP UDP 5835 +2010-12-01 23:52:14 Atlanta 10.0.0.1 62922 Texas 10.0.0.26 80 HTTP UDP 3704 +2010-12-01 23:52:15 Atlanta 10.0.0.1 50706 Texas 10.0.0.29 80 HTTP UDP 3398 +2010-12-01 23:52:16 Atlanta 10.0.0.1 50805 Texas 10.0.0.30 80 HTTP UDP 3144 +2010-12-01 23:52:17 Atlanta 10.0.0.1 65461 Texas 10.0.0.25 80 HTTP UDP 4399 +2010-12-01 23:52:18 Atlanta 10.0.0.1 65041 Texas 10.0.0.30 80 HTTP UDP 5952 +2010-12-01 23:52:19 Atlanta 10.0.0.1 65139 Texas 10.0.0.26 80 HTTP UDP 2681 +2010-12-01 23:52:20 Atlanta 10.0.0.1 61609 Texas 10.0.0.26 80 HTTP UDP 5016 +2010-12-01 23:52:21 Atlanta 10.0.0.1 51473 Texas 10.0.0.25 80 HTTP UDP 2656 +2010-12-01 23:52:22 Atlanta 10.0.0.1 58323 Texas 10.0.0.27 80 HTTP UDP 5014 +2010-12-01 23:52:23 Atlanta 10.0.0.1 54271 Texas 10.0.0.26 80 HTTP UDP 3054 +2010-12-01 23:52:24 Atlanta 10.0.0.1 63279 Texas 10.0.0.26 80 HTTP UDP 5763 +2010-12-01 23:52:25 Atlanta 10.0.0.1 64166 Texas 10.0.0.27 80 HTTP UDP 4416 +2010-12-01 23:52:26 Atlanta 10.0.0.1 61850 Texas 10.0.0.25 80 HTTP UDP 5343 +2010-12-01 23:52:27 Atlanta 10.0.0.1 53035 Texas 10.0.0.27 80 HTTP UDP 2975 +2010-12-01 23:52:28 Atlanta 10.0.0.1 55847 Texas 10.0.0.27 80 HTTP UDP 4102 +2010-12-01 23:52:29 Atlanta 10.0.0.1 57953 Texas 10.0.0.27 80 HTTP UDP 3365 +2010-12-01 23:52:30 Atlanta 10.0.0.1 64471 Texas 10.0.0.28 80 HTTP UDP 4358 +2010-12-01 23:52:31 Atlanta 10.0.0.1 57864 Texas 10.0.0.26 80 HTTP UDP 4359 +2010-12-01 23:52:32 Atlanta 10.0.0.1 51778 Texas 10.0.0.26 80 HTTP UDP 2391 +2010-12-01 23:52:33 Atlanta 10.0.0.1 59348 Texas 10.0.0.27 80 HTTP UDP 2855 +2010-12-01 23:52:34 Atlanta 10.0.0.1 62731 Texas 10.0.0.28 80 HTTP UDP 5016 +2010-12-01 23:52:35 Atlanta 10.0.0.1 64078 Texas 10.0.0.30 80 HTTP UDP 4868 +2010-12-01 23:52:36 Atlanta 10.0.0.1 59440 Texas 10.0.0.25 80 HTTP UDP 4179 +2010-12-01 23:52:37 Atlanta 10.0.0.1 64281 Texas 10.0.0.28 80 HTTP UDP 5273 +2010-12-01 23:52:38 Atlanta 10.0.0.1 49807 Texas 10.0.0.30 80 HTTP UDP 5868 +2010-12-01 23:52:39 Atlanta 10.0.0.1 57754 Texas 10.0.0.28 80 HTTP UDP 2135 +2010-12-01 23:52:40 Atlanta 10.0.0.1 61282 Texas 10.0.0.27 80 HTTP UDP 3135 +2010-12-01 23:52:41 Atlanta 10.0.0.1 60252 Texas 10.0.0.26 80 HTTP UDP 5160 +2010-12-01 23:52:42 Atlanta 10.0.0.1 60960 Texas 10.0.0.27 80 HTTP UDP 4381 +2010-12-01 23:52:43 Atlanta 10.0.0.1 51842 Texas 10.0.0.29 80 HTTP UDP 5480 +2010-12-01 23:52:44 Atlanta 10.0.0.1 54558 Texas 10.0.0.29 80 HTTP UDP 3198 +2010-12-01 23:52:45 Atlanta 10.0.0.1 51185 Texas 10.0.0.26 80 HTTP UDP 3156 +2010-12-01 23:52:46 Atlanta 10.0.0.1 50004 Texas 10.0.0.29 80 HTTP UDP 4193 +2010-12-01 23:52:47 Atlanta 10.0.0.1 50109 Texas 10.0.0.27 80 HTTP UDP 3979 +2010-12-01 23:52:48 Atlanta 10.0.0.1 51668 Texas 10.0.0.27 80 HTTP UDP 2577 +2010-12-01 23:52:49 Atlanta 10.0.0.1 59195 Texas 10.0.0.25 80 HTTP UDP 2818 +2010-12-01 23:52:50 Atlanta 10.0.0.1 51177 Texas 10.0.0.26 80 HTTP UDP 3267 +2010-12-01 23:52:51 Atlanta 10.0.0.1 61282 Texas 10.0.0.26 80 HTTP UDP 3781 +2010-12-01 23:52:52 Atlanta 10.0.0.1 51551 Texas 10.0.0.29 80 HTTP UDP 4093 +2010-12-01 23:52:53 Atlanta 10.0.0.1 50183 Texas 10.0.0.25 80 HTTP UDP 5545 +2010-12-01 23:52:54 Atlanta 10.0.0.1 55021 Texas 10.0.0.27 80 HTTP UDP 3940 +2010-12-01 23:52:55 Atlanta 10.0.0.1 61135 Texas 10.0.0.29 80 HTTP UDP 3990 +2010-12-01 23:52:56 Atlanta 10.0.0.1 59640 Texas 10.0.0.25 80 HTTP UDP 4600 +2010-12-01 23:52:57 Atlanta 10.0.0.1 57693 Texas 10.0.0.25 80 HTTP UDP 3671 +2010-12-01 23:52:58 Atlanta 10.0.0.1 61495 Texas 10.0.0.25 80 HTTP UDP 2192 +2010-12-01 23:52:59 Atlanta 10.0.0.1 58818 Texas 10.0.0.25 80 HTTP UDP 2602 +2010-12-01 23:53:00 Atlanta 10.0.0.1 51847 Texas 10.0.0.25 80 HTTP UDP 5016 +2010-12-01 23:53:01 Atlanta 10.0.0.1 55784 Texas 10.0.0.30 80 HTTP UDP 5662 +2010-12-01 23:53:02 Atlanta 10.0.0.1 51839 Texas 10.0.0.28 80 HTTP UDP 2135 +2010-12-01 23:53:03 Atlanta 10.0.0.1 64848 Texas 10.0.0.29 80 HTTP UDP 3390 +2010-12-01 23:53:04 Atlanta 10.0.0.1 58740 Texas 10.0.0.26 80 HTTP UDP 2477 +2010-12-01 23:53:05 Atlanta 10.0.0.1 54520 Texas 10.0.0.26 80 HTTP UDP 2947 +2010-12-01 23:53:06 Atlanta 10.0.0.1 61033 Texas 10.0.0.28 80 HTTP UDP 3918 +2010-12-01 23:53:07 Atlanta 10.0.0.1 52800 Texas 10.0.0.30 80 HTTP UDP 3096 +2010-12-01 23:53:08 Atlanta 10.0.0.1 50557 Texas 10.0.0.28 80 HTTP UDP 3079 +2010-12-01 23:53:09 Atlanta 10.0.0.1 64494 Texas 10.0.0.30 80 HTTP UDP 4260 +2010-12-01 23:53:10 Atlanta 10.0.0.1 59572 Texas 10.0.0.29 80 HTTP UDP 5966 +2010-12-01 23:53:11 Atlanta 10.0.0.1 50475 Texas 10.0.0.25 80 HTTP UDP 4545 +2010-12-01 23:53:12 Atlanta 10.0.0.1 60727 Texas 10.0.0.30 80 HTTP UDP 2563 +2010-12-01 23:53:13 Atlanta 10.0.0.1 65165 Texas 10.0.0.28 80 HTTP UDP 5940 +2010-12-01 23:53:14 Atlanta 10.0.0.1 63143 Texas 10.0.0.28 80 HTTP UDP 3790 +2010-12-01 23:53:15 Atlanta 10.0.0.1 61922 Texas 10.0.0.29 80 HTTP UDP 5482 +2010-12-01 23:53:16 Atlanta 10.0.0.1 50588 Texas 10.0.0.30 80 HTTP UDP 2135 +2010-12-01 23:53:17 Atlanta 10.0.0.1 56088 Texas 10.0.0.28 80 HTTP UDP 5366 +2010-12-01 23:53:18 Atlanta 10.0.0.1 55089 Texas 10.0.0.27 80 HTTP UDP 4090 +2010-12-01 23:53:19 Atlanta 10.0.0.1 57741 Texas 10.0.0.26 80 HTTP UDP 5849 +2010-12-01 23:53:20 Atlanta 10.0.0.1 54824 Texas 10.0.0.30 80 HTTP UDP 5284 +2010-12-01 23:53:21 Atlanta 10.0.0.1 61912 Texas 10.0.0.30 80 HTTP UDP 3581 +2010-12-01 23:53:22 Atlanta 10.0.0.1 51508 Texas 10.0.0.26 80 HTTP UDP 2215 +2010-12-01 23:53:23 Atlanta 10.0.0.1 62038 Texas 10.0.0.30 80 HTTP UDP 3138 +2010-12-01 23:53:24 Atlanta 10.0.0.1 56595 Texas 10.0.0.30 80 HTTP UDP 4887 +2010-12-01 23:53:25 Atlanta 10.0.0.1 62856 Texas 10.0.0.26 80 HTTP UDP 5675 +2010-12-01 23:53:26 Atlanta 10.0.0.1 52014 Texas 10.0.0.28 80 HTTP UDP 3066 +2010-12-01 23:53:27 Atlanta 10.0.0.1 59955 Texas 10.0.0.26 80 HTTP UDP 5584 +2010-12-01 23:53:28 Atlanta 10.0.0.1 59316 Texas 10.0.0.27 80 HTTP UDP 5500 +2010-12-01 23:53:29 Atlanta 10.0.0.1 52756 Texas 10.0.0.27 80 HTTP UDP 2790 +2010-12-01 23:53:30 Atlanta 10.0.0.1 65466 Texas 10.0.0.30 80 HTTP UDP 5952 +2010-12-01 23:53:31 Atlanta 10.0.0.1 49464 Texas 10.0.0.29 80 HTTP UDP 3096 +2010-12-01 23:53:32 Atlanta 10.0.0.1 59091 Texas 10.0.0.27 80 HTTP UDP 5204 +2010-12-01 23:53:33 Atlanta 10.0.0.1 59012 Texas 10.0.0.30 80 HTTP UDP 5603 +2010-12-01 23:53:34 Atlanta 10.0.0.1 50285 Texas 10.0.0.28 80 HTTP UDP 3326 +2010-12-01 23:53:35 Atlanta 10.0.0.1 58726 Texas 10.0.0.26 80 HTTP UDP 4359 +2010-12-01 23:53:36 Atlanta 10.0.0.1 63108 Texas 10.0.0.30 80 HTTP UDP 4804 +2010-12-01 23:53:37 Atlanta 10.0.0.1 62633 Texas 10.0.0.28 80 HTTP UDP 5817 +2010-12-01 23:53:38 Atlanta 10.0.0.1 58884 Texas 10.0.0.26 80 HTTP UDP 4163 +2010-12-01 23:53:39 Atlanta 10.0.0.1 62098 Texas 10.0.0.26 80 HTTP UDP 2249 +2010-12-01 23:53:40 Atlanta 10.0.0.1 57896 Texas 10.0.0.29 80 HTTP UDP 5002 +2010-12-01 23:53:41 Atlanta 10.0.0.1 55564 Texas 10.0.0.26 80 HTTP UDP 5422 +2010-12-01 23:53:42 Atlanta 10.0.0.1 64537 Texas 10.0.0.25 80 HTTP UDP 4053 +2010-12-01 23:53:43 Atlanta 10.0.0.1 52271 Texas 10.0.0.27 80 HTTP UDP 4636 +2010-12-01 23:53:44 Atlanta 10.0.0.1 65216 Texas 10.0.0.30 80 HTTP UDP 2001 +2010-12-01 23:53:45 Atlanta 10.0.0.1 62460 Texas 10.0.0.25 80 HTTP UDP 2997 +2010-12-01 23:53:46 Atlanta 10.0.0.1 54293 Texas 10.0.0.25 80 HTTP UDP 3107 +2010-12-01 23:53:47 Atlanta 10.0.0.1 49596 Texas 10.0.0.30 80 HTTP UDP 4918 +2010-12-01 23:53:48 Atlanta 10.0.0.1 64395 Texas 10.0.0.28 80 HTTP UDP 3302 +2010-12-01 23:53:49 Atlanta 10.0.0.1 57110 Texas 10.0.0.28 80 HTTP UDP 2882 +2010-12-01 23:53:50 Atlanta 10.0.0.1 53258 Texas 10.0.0.28 80 HTTP UDP 5966 +2010-12-01 23:53:51 Atlanta 10.0.0.1 61392 Texas 10.0.0.29 80 HTTP UDP 5020 +2010-12-01 23:53:52 Atlanta 10.0.0.1 55377 Texas 10.0.0.25 80 HTTP UDP 4576 +2010-12-01 23:53:53 Atlanta 10.0.0.1 63356 Texas 10.0.0.27 80 HTTP UDP 4032 +2010-12-01 23:53:54 Atlanta 10.0.0.1 63913 Texas 10.0.0.26 80 HTTP UDP 4817 +2010-12-01 23:53:55 Atlanta 10.0.0.1 57334 Texas 10.0.0.27 80 HTTP UDP 5047 +2010-12-01 23:53:56 Atlanta 10.0.0.1 50036 Texas 10.0.0.25 80 HTTP UDP 2985 +2010-12-01 23:53:57 Atlanta 10.0.0.1 56594 Texas 10.0.0.28 80 HTTP UDP 2355 +2010-12-01 23:53:58 Atlanta 10.0.0.1 49822 Texas 10.0.0.25 80 HTTP UDP 4886 +2010-12-01 23:53:59 Atlanta 10.0.0.1 52229 Texas 10.0.0.30 80 HTTP UDP 2332 +2010-12-01 23:54:00 Atlanta 10.0.0.1 55823 Texas 10.0.0.28 80 HTTP UDP 5252 +2010-12-01 23:54:01 Atlanta 10.0.0.1 60926 Texas 10.0.0.27 80 HTTP UDP 2627 +2010-12-01 23:54:02 Atlanta 10.0.0.1 50363 Texas 10.0.0.27 80 HTTP UDP 4324 +2010-12-01 23:54:03 Atlanta 10.0.0.1 57559 Texas 10.0.0.28 80 HTTP UDP 3326 +2010-12-01 23:54:04 Atlanta 10.0.0.1 55156 Texas 10.0.0.27 80 HTTP UDP 3454 +2010-12-01 23:54:05 Atlanta 10.0.0.1 53641 Texas 10.0.0.26 80 HTTP UDP 4933 +2010-12-01 23:54:06 Atlanta 10.0.0.1 49903 Texas 10.0.0.25 80 HTTP UDP 3032 +2010-12-01 23:54:07 Atlanta 10.0.0.1 62124 Texas 10.0.0.30 80 HTTP UDP 2830 +2010-12-01 23:54:08 Atlanta 10.0.0.1 60108 Texas 10.0.0.25 80 HTTP UDP 4864 +2010-12-01 23:54:09 Atlanta 10.0.0.1 55283 Texas 10.0.0.30 80 HTTP UDP 3476 +2010-12-01 23:54:10 Atlanta 10.0.0.1 65076 Texas 10.0.0.26 80 HTTP UDP 4794 +2010-12-01 23:54:11 Atlanta 10.0.0.1 54941 Texas 10.0.0.28 80 HTTP UDP 3345 +2010-12-01 23:54:12 Atlanta 10.0.0.1 64563 Texas 10.0.0.26 80 HTTP UDP 5347 +2010-12-01 23:54:13 Atlanta 10.0.0.1 61606 Texas 10.0.0.28 80 HTTP UDP 4868 +2010-12-01 23:54:14 Atlanta 10.0.0.1 54937 Texas 10.0.0.26 80 HTTP UDP 2985 +2010-12-01 23:54:15 Atlanta 10.0.0.1 50237 Texas 10.0.0.27 80 HTTP UDP 2757 +2010-12-01 23:54:16 Atlanta 10.0.0.1 59700 Texas 10.0.0.25 80 HTTP UDP 5389 +2010-12-01 23:54:17 Atlanta 10.0.0.1 55361 Texas 10.0.0.28 80 HTTP UDP 5694 +2010-12-01 23:54:18 Atlanta 10.0.0.1 61025 Texas 10.0.0.28 80 HTTP UDP 2891 +2010-12-01 23:54:19 Atlanta 10.0.0.1 65247 Texas 10.0.0.26 80 HTTP UDP 2577 +2010-12-01 23:54:20 Atlanta 10.0.0.1 58036 Texas 10.0.0.25 80 HTTP UDP 2526 +2010-12-01 23:54:21 Atlanta 10.0.0.1 52435 Texas 10.0.0.25 80 HTTP UDP 2562 +2010-12-01 23:54:22 Atlanta 10.0.0.1 55277 Texas 10.0.0.25 80 HTTP UDP 3122 +2010-12-01 23:54:23 Atlanta 10.0.0.1 54035 Texas 10.0.0.30 80 HTTP UDP 2511 +2010-12-01 23:54:24 Atlanta 10.0.0.1 63964 Texas 10.0.0.26 80 HTTP UDP 3572 +2010-12-01 23:54:25 Atlanta 10.0.0.1 55999 Texas 10.0.0.26 80 HTTP UDP 4162 +2010-12-01 23:54:26 Atlanta 10.0.0.1 52002 Texas 10.0.0.29 80 HTTP UDP 5918 +2010-12-01 23:54:27 Atlanta 10.0.0.1 51879 Texas 10.0.0.28 80 HTTP UDP 5039 +2010-12-01 23:54:28 Atlanta 10.0.0.1 52060 Texas 10.0.0.29 80 HTTP UDP 5011 +2010-12-01 23:54:29 Atlanta 10.0.0.1 56041 Texas 10.0.0.29 80 HTTP UDP 3628 +2010-12-01 23:54:30 Atlanta 10.0.0.1 52774 Texas 10.0.0.26 80 HTTP UDP 3947 +2010-12-01 23:54:31 Atlanta 10.0.0.1 62657 Texas 10.0.0.30 80 HTTP UDP 2921 +2010-12-01 23:54:32 Atlanta 10.0.0.1 59272 Texas 10.0.0.30 80 HTTP UDP 3934 +2010-12-01 23:54:33 Atlanta 10.0.0.1 57146 Texas 10.0.0.28 80 HTTP UDP 4598 +2010-12-01 23:54:34 Atlanta 10.0.0.1 55285 Texas 10.0.0.25 80 HTTP UDP 2642 +2010-12-01 23:54:35 Atlanta 10.0.0.1 52598 Texas 10.0.0.29 80 HTTP UDP 3125 +2010-12-01 23:54:36 Atlanta 10.0.0.1 58261 Texas 10.0.0.30 80 HTTP UDP 3585 +2010-12-01 23:54:37 Atlanta 10.0.0.1 58304 Texas 10.0.0.28 80 HTTP UDP 4657 +2010-12-01 23:54:38 Atlanta 10.0.0.1 65079 Texas 10.0.0.25 80 HTTP UDP 4138 +2010-12-01 23:54:39 Atlanta 10.0.0.1 63484 Texas 10.0.0.30 80 HTTP UDP 3025 +2010-12-01 23:54:40 Atlanta 10.0.0.1 50273 Texas 10.0.0.29 80 HTTP UDP 4724 +2010-12-01 23:54:41 Atlanta 10.0.0.1 59212 Texas 10.0.0.26 80 HTTP UDP 4903 +2010-12-01 23:54:42 Atlanta 10.0.0.1 55807 Texas 10.0.0.25 80 HTTP UDP 3538 +2010-12-01 23:54:43 Atlanta 10.0.0.1 63819 Texas 10.0.0.28 80 HTTP UDP 2594 +2010-12-01 23:54:44 Atlanta 10.0.0.1 51146 Texas 10.0.0.27 80 HTTP UDP 4411 +2010-12-01 23:54:45 Atlanta 10.0.0.1 64076 Texas 10.0.0.25 80 HTTP UDP 4407 +2010-12-01 23:54:46 Atlanta 10.0.0.1 58613 Texas 10.0.0.26 80 HTTP UDP 2553 +2010-12-01 23:54:47 Atlanta 10.0.0.1 57769 Texas 10.0.0.26 80 HTTP UDP 2523 +2010-12-01 23:54:48 Atlanta 10.0.0.1 60658 Texas 10.0.0.28 80 HTTP UDP 5326 +2010-12-01 23:54:49 Atlanta 10.0.0.1 55474 Texas 10.0.0.29 80 HTTP UDP 5098 +2010-12-01 23:54:50 Atlanta 10.0.0.1 51858 Texas 10.0.0.28 80 HTTP UDP 5620 +2010-12-01 23:54:51 Atlanta 10.0.0.1 51712 Texas 10.0.0.29 80 HTTP UDP 4744 +2010-12-01 23:54:52 Atlanta 10.0.0.1 63893 Texas 10.0.0.25 80 HTTP UDP 3860 +2010-12-01 23:54:53 Atlanta 10.0.0.1 51824 Texas 10.0.0.27 80 HTTP UDP 4564 +2010-12-01 23:54:54 Atlanta 10.0.0.1 60172 Texas 10.0.0.26 80 HTTP UDP 5016 +2010-12-01 23:54:55 Atlanta 10.0.0.1 57627 Texas 10.0.0.26 80 HTTP UDP 4521 +2010-12-01 23:54:56 Atlanta 10.0.0.1 65003 Texas 10.0.0.27 80 HTTP UDP 4113 +2010-12-01 23:54:57 Atlanta 10.0.0.1 62529 Texas 10.0.0.29 80 HTTP UDP 5628 +2010-12-01 23:54:58 Atlanta 10.0.0.1 61524 Texas 10.0.0.27 80 HTTP UDP 3241 +2010-12-01 23:54:59 Atlanta 10.0.0.1 64252 Texas 10.0.0.28 80 HTTP UDP 2579 +2010-12-01 23:55:00 Atlanta 10.0.0.1 50537 Texas 10.0.0.26 80 HTTP UDP 4397 +2010-12-01 23:55:01 Atlanta 10.0.0.1 63164 Texas 10.0.0.28 80 HTTP UDP 3872 +2010-12-01 23:55:02 Atlanta 10.0.0.1 53282 Texas 10.0.0.30 80 HTTP UDP 2579 +2010-12-01 23:55:03 Atlanta 10.0.0.1 49202 Texas 10.0.0.28 80 HTTP UDP 4179 +2010-12-01 23:55:04 Atlanta 10.0.0.1 55174 Texas 10.0.0.30 80 HTTP UDP 4545 +2010-12-01 23:55:05 Atlanta 10.0.0.1 50105 Texas 10.0.0.25 80 HTTP UDP 3196 +2010-12-01 23:55:06 Atlanta 10.0.0.1 63959 Texas 10.0.0.26 80 HTTP UDP 2298 +2010-12-01 23:55:07 Atlanta 10.0.0.1 49659 Texas 10.0.0.25 80 HTTP UDP 3398 +2010-12-01 23:55:08 Atlanta 10.0.0.1 51708 Texas 10.0.0.27 80 HTTP UDP 2690 +2010-12-01 23:55:09 Atlanta 10.0.0.1 59587 Texas 10.0.0.25 80 HTTP UDP 2891 +2010-12-01 23:55:10 Atlanta 10.0.0.1 51427 Texas 10.0.0.26 80 HTTP UDP 3357 +2010-12-01 23:55:11 Atlanta 10.0.0.1 63898 Texas 10.0.0.27 80 HTTP UDP 3063 +2010-12-01 23:55:12 Atlanta 10.0.0.1 50035 Texas 10.0.0.28 80 HTTP UDP 2877 +2010-12-01 23:55:13 Atlanta 10.0.0.1 54346 Texas 10.0.0.26 80 HTTP UDP 5336 +2010-12-01 23:55:14 Atlanta 10.0.0.1 53091 Texas 10.0.0.29 80 HTTP UDP 3135 +2010-12-01 23:55:15 Atlanta 10.0.0.1 61897 Texas 10.0.0.30 80 HTTP UDP 5092 +2010-12-01 23:55:16 Atlanta 10.0.0.1 57127 Texas 10.0.0.28 80 HTTP UDP 5556 +2010-12-01 23:55:17 Atlanta 10.0.0.1 57177 Texas 10.0.0.27 80 HTTP UDP 2819 +2010-12-01 23:55:18 Atlanta 10.0.0.1 54693 Texas 10.0.0.29 80 HTTP UDP 2813 +2010-12-01 23:55:19 Atlanta 10.0.0.1 52691 Texas 10.0.0.29 80 HTTP UDP 4121 +2010-12-01 23:55:20 Atlanta 10.0.0.1 50566 Texas 10.0.0.30 80 HTTP UDP 2523 +2010-12-01 23:55:21 Atlanta 10.0.0.1 56662 Texas 10.0.0.27 80 HTTP UDP 5243 +2010-12-01 23:55:22 Atlanta 10.0.0.1 61703 Texas 10.0.0.30 80 HTTP UDP 2231 +2010-12-01 23:55:23 Atlanta 10.0.0.1 57361 Texas 10.0.0.29 80 HTTP UDP 3189 +2010-12-01 23:55:24 Atlanta 10.0.0.1 59428 Texas 10.0.0.29 80 HTTP UDP 4381 +2010-12-01 23:55:25 Atlanta 10.0.0.1 58406 Texas 10.0.0.26 80 HTTP UDP 2906 +2010-12-01 23:55:26 Atlanta 10.0.0.1 56837 Texas 10.0.0.30 80 HTTP UDP 4849 +2010-12-01 23:55:27 Atlanta 10.0.0.1 51005 Texas 10.0.0.28 80 HTTP UDP 5011 +2010-12-01 23:55:28 Atlanta 10.0.0.1 53842 Texas 10.0.0.29 80 HTTP UDP 3657 +2010-12-01 23:55:29 Atlanta 10.0.0.1 52074 Texas 10.0.0.27 80 HTTP UDP 4936 +2010-12-01 23:55:30 Atlanta 10.0.0.1 50854 Texas 10.0.0.29 80 HTTP UDP 3223 +2010-12-01 23:55:31 Atlanta 10.0.0.1 53717 Texas 10.0.0.25 80 HTTP UDP 4935 +2010-12-01 23:55:32 Atlanta 10.0.0.1 63808 Texas 10.0.0.26 80 HTTP UDP 3042 +2010-12-01 23:55:33 Atlanta 10.0.0.1 49947 Texas 10.0.0.28 80 HTTP UDP 5935 +2010-12-01 23:55:34 Atlanta 10.0.0.1 52952 Texas 10.0.0.25 80 HTTP UDP 4193 +2010-12-01 23:55:35 Atlanta 10.0.0.1 53750 Texas 10.0.0.29 80 HTTP UDP 5231 +2010-12-01 23:55:36 Atlanta 10.0.0.1 58874 Texas 10.0.0.25 80 HTTP UDP 5668 +2010-12-01 23:55:37 Atlanta 10.0.0.1 64292 Texas 10.0.0.30 80 HTTP UDP 3704 +2010-12-01 23:55:38 Atlanta 10.0.0.1 63594 Texas 10.0.0.30 80 HTTP UDP 4369 +2010-12-01 23:55:39 Atlanta 10.0.0.1 58468 Texas 10.0.0.26 80 HTTP UDP 2391 +2010-12-01 23:55:40 Atlanta 10.0.0.1 58759 Texas 10.0.0.30 80 HTTP UDP 3013 +2010-12-01 23:55:41 Atlanta 10.0.0.1 58448 Texas 10.0.0.27 80 HTTP UDP 4753 +2010-12-01 23:55:42 Atlanta 10.0.0.1 58540 Texas 10.0.0.26 80 HTTP UDP 5675 +2010-12-01 23:55:43 Atlanta 10.0.0.1 60959 Texas 10.0.0.27 80 HTTP UDP 4600 +2010-12-01 23:55:44 Atlanta 10.0.0.1 63534 Texas 10.0.0.27 80 HTTP UDP 5552 +2010-12-01 23:55:45 Atlanta 10.0.0.1 63278 Texas 10.0.0.25 80 HTTP UDP 2022 +2010-12-01 23:55:46 Atlanta 10.0.0.1 56365 Texas 10.0.0.30 80 HTTP UDP 5826 +2010-12-01 23:55:47 Atlanta 10.0.0.1 62326 Texas 10.0.0.27 80 HTTP UDP 4589 +2010-12-01 23:55:48 Atlanta 10.0.0.1 56999 Texas 10.0.0.25 80 HTTP UDP 3788 +2010-12-01 23:55:49 Atlanta 10.0.0.1 64137 Texas 10.0.0.25 80 HTTP UDP 2954 +2010-12-01 23:55:50 Atlanta 10.0.0.1 49625 Texas 10.0.0.28 80 HTTP UDP 3544 +2010-12-01 23:55:51 Atlanta 10.0.0.1 52918 Texas 10.0.0.28 80 HTTP UDP 4379 +2010-12-01 23:55:52 Atlanta 10.0.0.1 54807 Texas 10.0.0.27 80 HTTP UDP 3786 +2010-12-01 23:55:53 Atlanta 10.0.0.1 54850 Texas 10.0.0.30 80 HTTP UDP 2627 +2010-12-01 23:55:54 Atlanta 10.0.0.1 50718 Texas 10.0.0.27 80 HTTP UDP 5959 +2010-12-01 23:55:55 Atlanta 10.0.0.1 56751 Texas 10.0.0.30 80 HTTP UDP 4521 +2010-12-01 23:55:56 Atlanta 10.0.0.1 59277 Texas 10.0.0.30 80 HTTP UDP 3205 +2010-12-01 23:55:57 Atlanta 10.0.0.1 53272 Texas 10.0.0.26 80 HTTP UDP 2611 +2010-12-01 23:55:58 Atlanta 10.0.0.1 59513 Texas 10.0.0.29 80 HTTP UDP 2067 +2010-12-01 23:55:59 Atlanta 10.0.0.1 52208 Texas 10.0.0.27 80 HTTP UDP 2480 +2010-12-01 23:56:00 Atlanta 10.0.0.1 53395 Texas 10.0.0.27 80 HTTP UDP 5543 +2010-12-01 23:56:01 Atlanta 10.0.0.1 57430 Texas 10.0.0.25 80 HTTP UDP 2312 +2010-12-01 23:56:02 Atlanta 10.0.0.1 54495 Texas 10.0.0.27 80 HTTP UDP 5422 +2010-12-01 23:56:03 Atlanta 10.0.0.1 58238 Texas 10.0.0.25 80 HTTP UDP 4868 +2010-12-01 23:56:04 Atlanta 10.0.0.1 55825 Texas 10.0.0.27 80 HTTP UDP 3899 +2010-12-01 23:56:05 Atlanta 10.0.0.1 52661 Texas 10.0.0.29 80 HTTP UDP 2940 +2010-12-01 23:56:06 Atlanta 10.0.0.1 51831 Texas 10.0.0.29 80 HTTP UDP 3156 +2010-12-01 23:56:07 Atlanta 10.0.0.1 50627 Texas 10.0.0.26 80 HTTP UDP 3572 +2010-12-01 23:56:08 Atlanta 10.0.0.1 60770 Texas 10.0.0.28 80 HTTP UDP 5472 +2010-12-01 23:56:09 Atlanta 10.0.0.1 51794 Texas 10.0.0.28 80 HTTP UDP 2782 +2010-12-01 23:56:10 Atlanta 10.0.0.1 60153 Texas 10.0.0.26 80 HTTP UDP 3572 +2010-12-01 23:56:11 Atlanta 10.0.0.1 57722 Texas 10.0.0.28 80 HTTP UDP 5749 +2010-12-01 23:56:12 Atlanta 10.0.0.1 49531 Texas 10.0.0.25 80 HTTP UDP 3054 +2010-12-01 23:56:13 Atlanta 10.0.0.1 56348 Texas 10.0.0.29 80 HTTP UDP 4316 +2010-12-01 23:56:14 Atlanta 10.0.0.1 54975 Texas 10.0.0.27 80 HTTP UDP 2595 +2010-12-01 23:56:15 Atlanta 10.0.0.1 60224 Texas 10.0.0.30 80 HTTP UDP 5197 +2010-12-01 23:56:16 Atlanta 10.0.0.1 60642 Texas 10.0.0.30 80 HTTP UDP 5963 +2010-12-01 23:56:17 Atlanta 10.0.0.1 58536 Texas 10.0.0.25 80 HTTP UDP 3241 +2010-12-01 23:56:18 Atlanta 10.0.0.1 52002 Texas 10.0.0.27 80 HTTP UDP 3970 +2010-12-01 23:56:19 Atlanta 10.0.0.1 50093 Texas 10.0.0.27 80 HTTP UDP 2059 +2010-12-01 23:56:20 Atlanta 10.0.0.1 61917 Texas 10.0.0.29 80 HTTP UDP 3733 +2010-12-01 23:56:21 Atlanta 10.0.0.1 55165 Texas 10.0.0.28 80 HTTP UDP 2572 +2010-12-01 23:56:22 Atlanta 10.0.0.1 57531 Texas 10.0.0.29 80 HTTP UDP 4061 +2010-12-01 23:56:23 Atlanta 10.0.0.1 63896 Texas 10.0.0.28 80 HTTP UDP 2554 +2010-12-01 23:56:24 Atlanta 10.0.0.1 57268 Texas 10.0.0.28 80 HTTP UDP 5309 +2010-12-01 23:56:25 Atlanta 10.0.0.1 52603 Texas 10.0.0.26 80 HTTP UDP 2732 +2010-12-01 23:56:26 Atlanta 10.0.0.1 57282 Texas 10.0.0.26 80 HTTP UDP 3545 +2010-12-01 23:56:27 Atlanta 10.0.0.1 65382 Texas 10.0.0.25 80 HTTP UDP 5593 +2010-12-01 23:56:28 Atlanta 10.0.0.1 59089 Texas 10.0.0.29 80 HTTP UDP 2545 +2010-12-01 23:56:29 Atlanta 10.0.0.1 51219 Texas 10.0.0.29 80 HTTP UDP 4359 +2010-12-01 23:56:30 Atlanta 10.0.0.1 56809 Texas 10.0.0.28 80 HTTP UDP 5849 +2010-12-01 23:56:31 Atlanta 10.0.0.1 52863 Texas 10.0.0.28 80 HTTP UDP 5963 +2010-12-01 23:56:32 Atlanta 10.0.0.1 64802 Texas 10.0.0.28 80 HTTP UDP 4415 +2010-12-01 23:56:33 Atlanta 10.0.0.1 62609 Texas 10.0.0.28 80 HTTP UDP 2732 +2010-12-01 23:56:34 Atlanta 10.0.0.1 59721 Texas 10.0.0.26 80 HTTP UDP 3042 +2010-12-01 23:56:35 Atlanta 10.0.0.1 57547 Texas 10.0.0.29 80 HTTP UDP 3416 +2010-12-01 23:56:36 Atlanta 10.0.0.1 61494 Texas 10.0.0.27 80 HTTP UDP 3028 +2010-12-01 23:56:37 Atlanta 10.0.0.1 62983 Texas 10.0.0.28 80 HTTP UDP 3230 +2010-12-01 23:56:38 Atlanta 10.0.0.1 60074 Texas 10.0.0.28 80 HTTP UDP 5868 +2010-12-01 23:56:39 Atlanta 10.0.0.1 50421 Texas 10.0.0.26 80 HTTP UDP 2050 +2010-12-01 23:56:40 Atlanta 10.0.0.1 50121 Texas 10.0.0.27 80 HTTP UDP 2403 +2010-12-01 23:56:41 Atlanta 10.0.0.1 55053 Texas 10.0.0.28 80 HTTP UDP 4620 +2010-12-01 23:56:42 Atlanta 10.0.0.1 55558 Texas 10.0.0.27 80 HTTP UDP 2249 +2010-12-01 23:56:43 Atlanta 10.0.0.1 55127 Texas 10.0.0.25 80 HTTP UDP 2532 +2010-12-01 23:56:44 Atlanta 10.0.0.1 64567 Texas 10.0.0.30 80 HTTP UDP 2595 +2010-12-01 23:56:45 Atlanta 10.0.0.1 59507 Texas 10.0.0.26 80 HTTP UDP 2594 +2010-12-01 23:56:46 Atlanta 10.0.0.1 59900 Texas 10.0.0.29 80 HTTP UDP 5811 +2010-12-01 23:56:47 Atlanta 10.0.0.1 53399 Texas 10.0.0.25 80 HTTP UDP 3673 +2010-12-01 23:56:48 Atlanta 10.0.0.1 49341 Texas 10.0.0.28 80 HTTP UDP 3450 +2010-12-01 23:56:49 Atlanta 10.0.0.1 61454 Texas 10.0.0.30 80 HTTP UDP 5594 +2010-12-01 23:56:50 Atlanta 10.0.0.1 58515 Texas 10.0.0.27 80 HTTP UDP 2947 +2010-12-01 23:56:51 Atlanta 10.0.0.1 65042 Texas 10.0.0.25 80 HTTP UDP 3959 +2010-12-01 23:56:52 Atlanta 10.0.0.1 57918 Texas 10.0.0.29 80 HTTP UDP 5673 +2010-12-01 23:56:53 Atlanta 10.0.0.1 54466 Texas 10.0.0.28 80 HTTP UDP 5543 +2010-12-01 23:56:54 Atlanta 10.0.0.1 57091 Texas 10.0.0.29 80 HTTP UDP 2532 +2010-12-01 23:56:55 Atlanta 10.0.0.1 59176 Texas 10.0.0.25 80 HTTP UDP 2038 +2010-12-01 23:56:56 Atlanta 10.0.0.1 59547 Texas 10.0.0.26 80 HTTP UDP 2822 +2010-12-01 23:56:57 Atlanta 10.0.0.1 59042 Texas 10.0.0.27 80 HTTP UDP 5706 +2010-12-01 23:56:58 Atlanta 10.0.0.1 54835 Texas 10.0.0.29 80 HTTP UDP 3153 +2010-12-01 23:56:59 Atlanta 10.0.0.1 51366 Texas 10.0.0.28 80 HTTP UDP 3873 +2010-12-01 23:57:00 Atlanta 10.0.0.1 61393 Texas 10.0.0.25 80 HTTP UDP 3460 +2010-12-01 23:57:01 Atlanta 10.0.0.1 58032 Texas 10.0.0.26 80 HTTP UDP 4647 +2010-12-01 23:57:02 Atlanta 10.0.0.1 61594 Texas 10.0.0.28 80 HTTP UDP 4369 +2010-12-01 23:57:03 Atlanta 10.0.0.1 50525 Texas 10.0.0.29 80 HTTP UDP 2125 +2010-12-01 23:57:04 Atlanta 10.0.0.1 52356 Texas 10.0.0.30 80 HTTP UDP 4501 +2010-12-01 23:57:05 Atlanta 10.0.0.1 63492 Texas 10.0.0.29 80 HTTP UDP 3764 +2010-12-01 23:57:06 Atlanta 10.0.0.1 54076 Texas 10.0.0.28 80 HTTP UDP 5309 +2010-12-01 23:57:07 Atlanta 10.0.0.1 57736 Texas 10.0.0.25 80 HTTP UDP 2595 +2010-12-01 23:57:08 Atlanta 10.0.0.1 53153 Texas 10.0.0.26 80 HTTP UDP 5284 +2010-12-01 23:57:09 Atlanta 10.0.0.1 57617 Texas 10.0.0.28 80 HTTP UDP 3009 +2010-12-01 23:57:10 Atlanta 10.0.0.1 61645 Texas 10.0.0.26 80 HTTP UDP 4058 +2010-12-01 23:57:11 Atlanta 10.0.0.1 61495 Texas 10.0.0.27 80 HTTP UDP 4369 +2010-12-01 23:57:12 Atlanta 10.0.0.1 50045 Texas 10.0.0.29 80 HTTP UDP 2378 +2010-12-01 23:57:13 Atlanta 10.0.0.1 50056 Texas 10.0.0.25 80 HTTP UDP 2940 +2010-12-01 23:57:14 Atlanta 10.0.0.1 59238 Texas 10.0.0.30 80 HTTP UDP 5774 +2010-12-01 23:57:15 Atlanta 10.0.0.1 54461 Texas 10.0.0.25 80 HTTP UDP 5162 +2010-12-01 23:57:16 Atlanta 10.0.0.1 58309 Texas 10.0.0.30 80 HTTP UDP 3996 +2010-12-01 23:57:17 Atlanta 10.0.0.1 59338 Texas 10.0.0.29 80 HTTP UDP 2318 +2010-12-01 23:57:18 Atlanta 10.0.0.1 51102 Texas 10.0.0.28 80 HTTP UDP 2294 +2010-12-01 23:57:19 Atlanta 10.0.0.1 57079 Texas 10.0.0.25 80 HTTP UDP 3261 +2010-12-01 23:57:20 Atlanta 10.0.0.1 55874 Texas 10.0.0.28 80 HTTP UDP 5035 +2010-12-01 23:57:21 Atlanta 10.0.0.1 59580 Texas 10.0.0.28 80 HTTP UDP 4506 +2010-12-01 23:57:22 Atlanta 10.0.0.1 60952 Texas 10.0.0.30 80 HTTP UDP 2689 +2010-12-01 23:57:23 Atlanta 10.0.0.1 60055 Texas 10.0.0.29 80 HTTP UDP 5336 +2010-12-01 23:57:24 Atlanta 10.0.0.1 52634 Texas 10.0.0.30 80 HTTP UDP 4041 +2010-12-01 23:57:25 Atlanta 10.0.0.1 63727 Texas 10.0.0.26 80 HTTP UDP 2867 +2010-12-01 23:57:26 Atlanta 10.0.0.1 54275 Texas 10.0.0.30 80 HTTP UDP 2319 +2010-12-01 23:57:27 Atlanta 10.0.0.1 62262 Texas 10.0.0.25 80 HTTP UDP 5997 +2010-12-01 23:57:28 Atlanta 10.0.0.1 64110 Texas 10.0.0.30 80 HTTP UDP 5468 +2010-12-01 23:57:29 Atlanta 10.0.0.1 58321 Texas 10.0.0.27 80 HTTP UDP 5256 +2010-12-01 23:57:30 Atlanta 10.0.0.1 61111 Texas 10.0.0.26 80 HTTP UDP 5194 +2010-12-01 23:57:31 Atlanta 10.0.0.1 60017 Texas 10.0.0.26 80 HTTP UDP 2012 +2010-12-01 23:57:32 Atlanta 10.0.0.1 49440 Texas 10.0.0.30 80 HTTP UDP 4356 +2010-12-01 23:57:33 Atlanta 10.0.0.1 62766 Texas 10.0.0.30 80 HTTP UDP 4855 +2010-12-01 23:57:34 Atlanta 10.0.0.1 57020 Texas 10.0.0.28 80 HTTP UDP 2640 +2010-12-01 23:57:35 Atlanta 10.0.0.1 55303 Texas 10.0.0.26 80 HTTP UDP 4229 +2010-12-01 23:57:36 Atlanta 10.0.0.1 54279 Texas 10.0.0.25 80 HTTP UDP 4903 +2010-12-01 23:57:37 Atlanta 10.0.0.1 49493 Texas 10.0.0.25 80 HTTP UDP 3450 +2010-12-01 23:57:38 Atlanta 10.0.0.1 53581 Texas 10.0.0.26 80 HTTP UDP 4319 +2010-12-01 23:57:39 Atlanta 10.0.0.1 59243 Texas 10.0.0.28 80 HTTP UDP 3522 +2010-12-01 23:57:40 Atlanta 10.0.0.1 62251 Texas 10.0.0.30 80 HTTP UDP 5810 +2010-12-01 23:57:41 Atlanta 10.0.0.1 55785 Texas 10.0.0.29 80 HTTP UDP 2329 +2010-12-01 23:57:42 Atlanta 10.0.0.1 55555 Texas 10.0.0.26 80 HTTP UDP 2295 +2010-12-01 23:57:43 Atlanta 10.0.0.1 49315 Texas 10.0.0.30 80 HTTP UDP 4177 +2010-12-01 23:57:44 Atlanta 10.0.0.1 51508 Texas 10.0.0.27 80 HTTP UDP 5565 +2010-12-01 23:57:45 Atlanta 10.0.0.1 50565 Texas 10.0.0.26 80 HTTP UDP 3789 +2010-12-01 23:57:46 Atlanta 10.0.0.1 52705 Texas 10.0.0.28 80 HTTP UDP 5828 +2010-12-01 23:57:47 Atlanta 10.0.0.1 51165 Texas 10.0.0.27 80 HTTP UDP 2313 +2010-12-01 23:57:48 Atlanta 10.0.0.1 65400 Texas 10.0.0.28 80 HTTP UDP 4241 +2010-12-01 23:57:49 Atlanta 10.0.0.1 58454 Texas 10.0.0.26 80 HTTP UDP 5092 +2010-12-01 23:57:50 Atlanta 10.0.0.1 50395 Texas 10.0.0.25 80 HTTP UDP 4369 +2010-12-01 23:57:51 Atlanta 10.0.0.1 64053 Texas 10.0.0.30 80 HTTP UDP 4414 +2010-12-01 23:57:52 Atlanta 10.0.0.1 64827 Texas 10.0.0.29 80 HTTP UDP 4777 +2010-12-01 23:57:53 Atlanta 10.0.0.1 59878 Texas 10.0.0.29 80 HTTP UDP 2728 +2010-12-01 23:57:54 Atlanta 10.0.0.1 60087 Texas 10.0.0.26 80 HTTP UDP 4936 +2010-12-01 23:57:55 Atlanta 10.0.0.1 49562 Texas 10.0.0.30 80 HTTP UDP 5997 +2010-12-01 23:57:56 Atlanta 10.0.0.1 63849 Texas 10.0.0.26 80 HTTP UDP 5501 +2010-12-01 23:57:57 Atlanta 10.0.0.1 55180 Texas 10.0.0.25 80 HTTP UDP 4977 +2010-12-01 23:57:58 Atlanta 10.0.0.1 60237 Texas 10.0.0.28 80 HTTP UDP 2595 +2010-12-01 23:57:59 Atlanta 10.0.0.1 55314 Texas 10.0.0.25 80 HTTP UDP 3034 +2010-12-01 23:58:00 Atlanta 10.0.0.1 59935 Texas 10.0.0.28 80 HTTP UDP 4969 +2010-12-01 23:58:01 Atlanta 10.0.0.1 51530 Texas 10.0.0.25 80 HTTP UDP 4172 +2010-12-01 23:58:02 Atlanta 10.0.0.1 56643 Texas 10.0.0.25 80 HTTP UDP 3905 +2010-12-01 23:58:03 Atlanta 10.0.0.1 64844 Texas 10.0.0.26 80 HTTP UDP 4902 +2010-12-01 23:58:04 Atlanta 10.0.0.1 51045 Texas 10.0.0.25 80 HTTP UDP 5258 +2010-12-01 23:58:05 Atlanta 10.0.0.1 56028 Texas 10.0.0.30 80 HTTP UDP 3959 +2010-12-01 23:58:06 Atlanta 10.0.0.1 51903 Texas 10.0.0.25 80 HTTP UDP 5705 +2010-12-01 23:58:07 Atlanta 10.0.0.1 55653 Texas 10.0.0.27 80 HTTP UDP 2144 +2010-12-01 23:58:08 Atlanta 10.0.0.1 61045 Texas 10.0.0.25 80 HTTP UDP 2734 +2010-12-01 23:58:09 Atlanta 10.0.0.1 50955 Texas 10.0.0.25 80 HTTP UDP 4849 +2010-12-01 23:58:10 Atlanta 10.0.0.1 53962 Texas 10.0.0.26 80 HTTP UDP 4561 +2010-12-01 23:58:11 Atlanta 10.0.0.1 58276 Texas 10.0.0.30 80 HTTP UDP 2002 +2010-12-01 23:58:12 Atlanta 10.0.0.1 52776 Texas 10.0.0.30 80 HTTP UDP 5806 +2010-12-01 23:58:13 Atlanta 10.0.0.1 52390 Texas 10.0.0.26 80 HTTP UDP 5652 +2010-12-01 23:58:14 Atlanta 10.0.0.1 53571 Texas 10.0.0.25 80 HTTP UDP 5570 +2010-12-01 23:58:15 Atlanta 10.0.0.1 51011 Texas 10.0.0.29 80 HTTP UDP 4751 +2010-12-01 23:58:16 Atlanta 10.0.0.1 62718 Texas 10.0.0.29 80 HTTP UDP 3669 +2010-12-01 23:58:17 Atlanta 10.0.0.1 52491 Texas 10.0.0.25 80 HTTP UDP 5002 +2010-12-01 23:58:18 Atlanta 10.0.0.1 64067 Texas 10.0.0.28 80 HTTP UDP 3990 +2010-12-01 23:58:19 Atlanta 10.0.0.1 49291 Texas 10.0.0.26 80 HTTP UDP 4163 +2010-12-01 23:58:20 Atlanta 10.0.0.1 55880 Texas 10.0.0.27 80 HTTP UDP 5397 +2010-12-01 23:58:21 Atlanta 10.0.0.1 63794 Texas 10.0.0.29 80 HTTP UDP 2882 +2010-12-01 23:58:22 Atlanta 10.0.0.1 63819 Texas 10.0.0.29 80 HTTP UDP 4753 +2010-12-01 23:58:23 Atlanta 10.0.0.1 54759 Texas 10.0.0.26 80 HTTP UDP 3160 +2010-12-01 23:58:24 Atlanta 10.0.0.1 64568 Texas 10.0.0.25 80 HTTP UDP 5482 +2010-12-01 23:58:25 Atlanta 10.0.0.1 65032 Texas 10.0.0.30 80 HTTP UDP 2186 +2010-12-01 23:58:26 Atlanta 10.0.0.1 63941 Texas 10.0.0.30 80 HTTP UDP 3202 +2010-12-01 23:58:27 Atlanta 10.0.0.1 61759 Texas 10.0.0.29 80 HTTP UDP 2311 +2010-12-01 23:58:28 Atlanta 10.0.0.1 62807 Texas 10.0.0.26 80 HTTP UDP 5163 +2010-12-01 23:58:29 Atlanta 10.0.0.1 53873 Texas 10.0.0.28 80 HTTP UDP 5422 +2010-12-01 23:58:30 Atlanta 10.0.0.1 58915 Texas 10.0.0.25 80 HTTP UDP 3704 +2010-12-01 23:58:31 Atlanta 10.0.0.1 49838 Texas 10.0.0.28 80 HTTP UDP 3034 +2010-12-01 23:58:32 Atlanta 10.0.0.1 61980 Texas 10.0.0.25 80 HTTP UDP 2805 +2010-12-01 23:58:33 Atlanta 10.0.0.1 51938 Texas 10.0.0.28 80 HTTP UDP 2906 +2010-12-01 23:58:34 Atlanta 10.0.0.1 49656 Texas 10.0.0.29 80 HTTP UDP 5224 +2010-12-01 23:58:35 Atlanta 10.0.0.1 49813 Texas 10.0.0.27 80 HTTP UDP 5223 +2010-12-01 23:58:36 Atlanta 10.0.0.1 60940 Texas 10.0.0.30 80 HTTP UDP 3826 +2010-12-01 23:58:37 Atlanta 10.0.0.1 65337 Texas 10.0.0.25 80 HTTP UDP 3343 +2010-12-01 23:58:38 Atlanta 10.0.0.1 60814 Texas 10.0.0.26 80 HTTP UDP 4006 +2010-12-01 23:58:39 Atlanta 10.0.0.1 62794 Texas 10.0.0.26 80 HTTP UDP 2818 +2010-12-01 23:58:40 Atlanta 10.0.0.1 62984 Texas 10.0.0.29 80 HTTP UDP 2594 +2010-12-01 23:58:41 Atlanta 10.0.0.1 64702 Texas 10.0.0.27 80 HTTP UDP 5347 +2010-12-01 23:58:42 Atlanta 10.0.0.1 52726 Texas 10.0.0.28 80 HTTP UDP 2804 +2010-12-01 23:58:43 Atlanta 10.0.0.1 52765 Texas 10.0.0.28 80 HTTP UDP 2701 +2010-12-01 23:58:44 Atlanta 10.0.0.1 54360 Texas 10.0.0.30 80 HTTP UDP 5348 +2010-12-01 23:58:45 Atlanta 10.0.0.1 57574 Texas 10.0.0.27 80 HTTP UDP 2746 +2010-12-01 23:58:46 Atlanta 10.0.0.1 52012 Texas 10.0.0.26 80 HTTP UDP 5919 +2010-12-01 23:58:47 Atlanta 10.0.0.1 57997 Texas 10.0.0.27 80 HTTP UDP 5930 +2010-12-01 23:58:48 Atlanta 10.0.0.1 51900 Texas 10.0.0.29 80 HTTP UDP 4777 +2010-12-01 23:58:49 Atlanta 10.0.0.1 56409 Texas 10.0.0.28 80 HTTP UDP 4969 +2010-12-01 23:58:50 Atlanta 10.0.0.1 57912 Texas 10.0.0.27 80 HTTP UDP 3764 +2010-12-01 23:58:51 Atlanta 10.0.0.1 62478 Texas 10.0.0.28 80 HTTP UDP 5221 +2010-12-01 23:58:52 Atlanta 10.0.0.1 62603 Texas 10.0.0.26 80 HTTP UDP 5668 +2010-12-01 23:58:53 Atlanta 10.0.0.1 51694 Texas 10.0.0.25 80 HTTP UDP 3067 +2010-12-01 23:58:54 Atlanta 10.0.0.1 60701 Texas 10.0.0.27 80 HTTP UDP 5545 +2010-12-01 23:58:55 Atlanta 10.0.0.1 50110 Texas 10.0.0.30 80 HTTP UDP 4273 +2010-12-01 23:58:56 Atlanta 10.0.0.1 63097 Texas 10.0.0.26 80 HTTP UDP 3079 +2010-12-01 23:58:57 Atlanta 10.0.0.1 63579 Texas 10.0.0.25 80 HTTP UDP 3597 +2010-12-01 23:58:58 Atlanta 10.0.0.1 57243 Texas 10.0.0.29 80 HTTP UDP 4297 +2010-12-01 23:58:59 Atlanta 10.0.0.1 56111 Texas 10.0.0.25 80 HTTP UDP 3398 +2010-12-01 23:59:00 Atlanta 10.0.0.1 61341 Texas 10.0.0.26 80 HTTP UDP 2257 +2010-12-01 23:59:01 Atlanta 10.0.0.1 57600 Texas 10.0.0.29 80 HTTP UDP 2024 +2010-12-01 23:59:02 Atlanta 10.0.0.1 58463 Texas 10.0.0.29 80 HTTP UDP 3997 +2010-12-01 23:59:03 Atlanta 10.0.0.1 53508 Texas 10.0.0.28 80 HTTP UDP 4064 +2010-12-01 23:59:04 Atlanta 10.0.0.1 62940 Texas 10.0.0.25 80 HTTP UDP 5853 +2010-12-01 23:59:05 Atlanta 10.0.0.1 56949 Texas 10.0.0.29 80 HTTP UDP 3139 +2010-12-01 23:59:06 Atlanta 10.0.0.1 52227 Texas 10.0.0.26 80 HTTP UDP 4934 +2010-12-01 23:59:07 Atlanta 10.0.0.1 52566 Texas 10.0.0.28 80 HTTP UDP 3497 +2010-12-01 23:59:08 Atlanta 10.0.0.1 62109 Texas 10.0.0.29 80 HTTP UDP 2046 +2010-12-01 23:59:09 Atlanta 10.0.0.1 50566 Texas 10.0.0.28 80 HTTP UDP 2154 +2010-12-01 23:59:10 Atlanta 10.0.0.1 64703 Texas 10.0.0.29 80 HTTP UDP 3121 +2010-12-01 23:59:11 Atlanta 10.0.0.1 61671 Texas 10.0.0.28 80 HTTP UDP 3945 +2010-12-01 23:59:12 Atlanta 10.0.0.1 60016 Texas 10.0.0.26 80 HTTP UDP 2659 +2010-12-01 23:59:13 Atlanta 10.0.0.1 56950 Texas 10.0.0.27 80 HTTP UDP 5594 +2010-12-01 23:59:14 Atlanta 10.0.0.1 52109 Texas 10.0.0.26 80 HTTP UDP 3223 +2010-12-01 23:59:15 Atlanta 10.0.0.1 50386 Texas 10.0.0.27 80 HTTP UDP 3708 +2010-12-01 23:59:16 Atlanta 10.0.0.1 64394 Texas 10.0.0.27 80 HTTP UDP 3326 +2010-12-01 23:59:17 Atlanta 10.0.0.1 56569 Texas 10.0.0.29 80 HTTP UDP 4589 +2010-12-01 23:59:18 Atlanta 10.0.0.1 57309 Texas 10.0.0.27 80 HTTP UDP 3959 +2010-12-01 23:59:19 Atlanta 10.0.0.1 65193 Texas 10.0.0.26 80 HTTP UDP 2363 +2010-12-01 23:59:20 Atlanta 10.0.0.1 49517 Texas 10.0.0.30 80 HTTP UDP 3896 +2010-12-01 23:59:21 Atlanta 10.0.0.1 59072 Texas 10.0.0.26 80 HTTP UDP 5853 +2010-12-01 23:59:22 Atlanta 10.0.0.1 64009 Texas 10.0.0.26 80 HTTP UDP 2144 +2010-12-01 23:59:23 Atlanta 10.0.0.1 57255 Texas 10.0.0.26 80 HTTP UDP 5959 +2010-12-01 23:59:24 Atlanta 10.0.0.1 49770 Texas 10.0.0.29 80 HTTP UDP 2422 +2010-12-01 23:59:25 Atlanta 10.0.0.1 53485 Texas 10.0.0.28 80 HTTP UDP 3345 +2010-12-01 23:59:26 Atlanta 10.0.0.1 49759 Texas 10.0.0.27 80 HTTP UDP 3104 +2010-12-01 23:59:27 Atlanta 10.0.0.1 50519 Texas 10.0.0.26 80 HTTP UDP 4502 +2010-12-01 23:59:28 Atlanta 10.0.0.1 50275 Texas 10.0.0.30 80 HTTP UDP 3450 +2010-12-01 23:59:29 Atlanta 10.0.0.1 53998 Texas 10.0.0.30 80 HTTP UDP 4826 +2010-12-01 23:59:30 Atlanta 10.0.0.1 54587 Texas 10.0.0.26 80 HTTP UDP 3183 +2010-12-01 23:59:31 Atlanta 10.0.0.1 64630 Texas 10.0.0.27 80 HTTP UDP 2690 +2010-12-01 23:59:32 Atlanta 10.0.0.1 61553 Texas 10.0.0.26 80 HTTP UDP 3223 +2010-12-01 23:59:33 Atlanta 10.0.0.1 52845 Texas 10.0.0.26 80 HTTP UDP 3736 +2010-12-01 23:59:34 Atlanta 10.0.0.1 51888 Texas 10.0.0.28 80 HTTP UDP 3189 +2010-12-01 23:59:35 Atlanta 10.0.0.1 49338 Texas 10.0.0.25 80 HTTP UDP 2319 +2010-12-01 23:59:36 Atlanta 10.0.0.1 49883 Texas 10.0.0.29 80 HTTP UDP 3647 +2010-12-01 23:59:37 Atlanta 10.0.0.1 59516 Texas 10.0.0.30 80 HTTP UDP 2380 +2010-12-01 23:59:38 Atlanta 10.0.0.1 52433 Texas 10.0.0.30 80 HTTP UDP 3317 +2010-12-01 23:59:39 Atlanta 10.0.0.1 59967 Texas 10.0.0.26 80 HTTP UDP 5888 +2010-12-01 23:59:40 Atlanta 10.0.0.1 51623 Texas 10.0.0.29 80 HTTP UDP 4893 +2010-12-01 23:59:41 Atlanta 10.0.0.1 56157 Texas 10.0.0.26 80 HTTP UDP 2659 +2010-12-01 23:59:42 Atlanta 10.0.0.1 55696 Texas 10.0.0.26 80 HTTP UDP 5603 +2010-12-01 23:59:43 Atlanta 10.0.0.1 50814 Texas 10.0.0.29 80 HTTP UDP 5899 +2010-12-01 23:59:44 Atlanta 10.0.0.1 59400 Texas 10.0.0.29 80 HTTP UDP 4160 +2010-12-01 23:59:45 Atlanta 10.0.0.1 59067 Texas 10.0.0.29 80 HTTP UDP 5700 +2010-12-01 23:59:46 Atlanta 10.0.0.1 51830 Texas 10.0.0.30 80 HTTP UDP 5029 +2010-12-01 23:59:47 Atlanta 10.0.0.1 54481 Texas 10.0.0.27 80 HTTP UDP 4355 +2010-12-01 23:59:48 Atlanta 10.0.0.1 61352 Texas 10.0.0.27 80 HTTP UDP 2172 +2010-12-01 23:59:49 Atlanta 10.0.0.1 55384 Texas 10.0.0.27 80 HTTP UDP 3174 +2010-12-01 23:59:50 Atlanta 10.0.0.1 50733 Texas 10.0.0.28 80 HTTP UDP 2484 +2010-12-01 23:59:51 Atlanta 10.0.0.1 64264 Texas 10.0.0.29 80 HTTP UDP 5543 +2010-12-01 23:59:52 Atlanta 10.0.0.1 57826 Texas 10.0.0.27 80 HTTP UDP 5527 +2010-12-01 23:59:53 Atlanta 10.0.0.1 52448 Texas 10.0.0.27 80 HTTP UDP 4849 +2010-12-01 23:59:54 Atlanta 10.0.0.1 51113 Texas 10.0.0.26 80 HTTP UDP 2746 +2010-12-01 23:59:55 Atlanta 10.0.0.1 61284 Texas 10.0.0.30 80 HTTP UDP 3959 +2010-12-01 23:59:56 Atlanta 10.0.0.1 59560 Texas 10.0.0.29 80 HTTP UDP 5837 +2010-12-01 23:59:57 Atlanta 10.0.0.1 60090 Texas 10.0.0.29 80 HTTP UDP 3079 +2010-12-01 23:59:58 Atlanta 10.0.0.1 52586 Texas 10.0.0.30 80 HTTP UDP 3202 +2010-12-01 23:59:59 Atlanta 10.0.0.1 62789 Texas 10.0.0.25 80 HTTP UDP 4097 diff --git a/.gitbook/assets/interest_rates.csv b/.gitbook/assets/interest_rates.csv new file mode 100644 index 00000000..6c832fff --- /dev/null +++ b/.gitbook/assets/interest_rates.csv @@ -0,0 +1,6 @@ +credit_rating,min_rate,max_rate +POOR,4.7,5.2 +GOOD,3.5,3.8 +FAIR,3.6,3.9 +EXCELLENT,3.00,3.4 +BAD,4.9,5.7 \ No newline at end of file diff --git a/.gitbook/assets/item_label.gif b/.gitbook/assets/item_label.gif new file mode 100644 index 00000000..b835c100 Binary files /dev/null and b/.gitbook/assets/item_label.gif differ diff --git a/.gitbook/assets/job_schedule.gif b/.gitbook/assets/job_schedule.gif new file mode 100644 index 00000000..f017c239 Binary files /dev/null and b/.gitbook/assets/job_schedule.gif differ diff --git a/.gitbook/assets/limits (1).gif b/.gitbook/assets/limits (1).gif new file mode 100644 index 00000000..7d1e4242 Binary files /dev/null and b/.gitbook/assets/limits (1).gif differ diff --git a/.gitbook/assets/limits.gif b/.gitbook/assets/limits.gif new file mode 100644 index 00000000..7d1e4242 Binary files /dev/null and b/.gitbook/assets/limits.gif differ diff --git a/.gitbook/assets/list_view.gif b/.gitbook/assets/list_view.gif new file mode 100644 index 00000000..c9c573ce Binary files /dev/null and b/.gitbook/assets/list_view.gif differ diff --git a/.gitbook/assets/off_peak.gif b/.gitbook/assets/off_peak.gif new file mode 100644 index 00000000..f59c794f Binary files /dev/null and b/.gitbook/assets/off_peak.gif differ diff --git a/.gitbook/assets/olw-jobs.png b/.gitbook/assets/olw-jobs.png new file mode 100644 index 00000000..c539fd33 Binary files /dev/null and b/.gitbook/assets/olw-jobs.png differ diff --git a/.gitbook/assets/outlier-tuning.png b/.gitbook/assets/outlier-tuning.png new file mode 100644 index 00000000..ec8e40fa Binary files /dev/null and b/.gitbook/assets/outlier-tuning.png differ diff --git a/.gitbook/assets/outliercalibratemodal.gif b/.gitbook/assets/outliercalibratemodal.gif new file mode 100644 index 00000000..aa99be92 Binary files /dev/null and b/.gitbook/assets/outliercalibratemodal.gif differ diff --git a/.gitbook/assets/outliercalibratenavigate.gif b/.gitbook/assets/outliercalibratenavigate.gif new file mode 100644 index 00000000..03012ee5 Binary files /dev/null and b/.gitbook/assets/outliercalibratenavigate.gif differ diff --git a/.gitbook/assets/outliercalibratenextdatapoint (1).gif b/.gitbook/assets/outliercalibratenextdatapoint (1).gif new file mode 100644 index 00000000..440aca4b Binary files /dev/null and b/.gitbook/assets/outliercalibratenextdatapoint (1).gif differ diff --git a/.gitbook/assets/outliercalibratenextdatapoint.gif b/.gitbook/assets/outliercalibratenextdatapoint.gif new file mode 100644 index 00000000..440aca4b Binary files /dev/null and b/.gitbook/assets/outliercalibratenextdatapoint.gif differ diff --git a/.gitbook/assets/owl-agent (1) (1).png b/.gitbook/assets/owl-agent (1) (1).png new file mode 100644 index 00000000..688047ca Binary files /dev/null and b/.gitbook/assets/owl-agent (1) (1).png differ diff --git a/.gitbook/assets/owl-agent (1).png b/.gitbook/assets/owl-agent (1).png new file mode 100644 index 00000000..688047ca Binary files /dev/null and b/.gitbook/assets/owl-agent (1).png differ diff --git a/.gitbook/assets/owl-agent.png b/.gitbook/assets/owl-agent.png new file mode 100644 index 00000000..a274a71e Binary files /dev/null and b/.gitbook/assets/owl-agent.png differ diff --git a/.gitbook/assets/owl-agents.png b/.gitbook/assets/owl-agents.png new file mode 100644 index 00000000..71443e53 Binary files /dev/null and b/.gitbook/assets/owl-agents.png differ diff --git a/.gitbook/assets/owl-alert-email-server.png b/.gitbook/assets/owl-alert-email-server.png new file mode 100644 index 00000000..01b02bc7 Binary files /dev/null and b/.gitbook/assets/owl-alert-email-server.png differ diff --git a/.gitbook/assets/owl-alert-email.png b/.gitbook/assets/owl-alert-email.png new file mode 100644 index 00000000..9ed52258 Binary files /dev/null and b/.gitbook/assets/owl-alert-email.png differ diff --git a/.gitbook/assets/owl-apis.png b/.gitbook/assets/owl-apis.png new file mode 100644 index 00000000..30561e25 Binary files /dev/null and b/.gitbook/assets/owl-apis.png differ diff --git a/.gitbook/assets/owl-assignment-q-MS_DB#1.png b/.gitbook/assets/owl-assignment-q-MS_DB#1.png new file mode 100644 index 00000000..bd7700f7 Binary files /dev/null and b/.gitbook/assets/owl-assignment-q-MS_DB#1.png differ diff --git a/.gitbook/assets/owl-assignment-q.pdf b/.gitbook/assets/owl-assignment-q.pdf new file mode 100644 index 00000000..31239948 Binary files /dev/null and b/.gitbook/assets/owl-assignment-q.pdf differ diff --git a/.gitbook/assets/owl-behavior-score.png b/.gitbook/assets/owl-behavior-score.png new file mode 100644 index 00000000..b1d552f3 Binary files /dev/null and b/.gitbook/assets/owl-behavior-score.png differ diff --git a/.gitbook/assets/owl-catalog-pii (1).png b/.gitbook/assets/owl-catalog-pii (1).png new file mode 100644 index 00000000..749c4982 Binary files /dev/null and b/.gitbook/assets/owl-catalog-pii (1).png differ diff --git a/.gitbook/assets/owl-catalog-pii (2).png b/.gitbook/assets/owl-catalog-pii (2).png new file mode 100644 index 00000000..5d5fe88e Binary files /dev/null and b/.gitbook/assets/owl-catalog-pii (2).png differ diff --git a/.gitbook/assets/owl-catalog-pii.png b/.gitbook/assets/owl-catalog-pii.png new file mode 100644 index 00000000..547c0281 Binary files /dev/null and b/.gitbook/assets/owl-catalog-pii.png differ diff --git a/.gitbook/assets/owl-catalog.png b/.gitbook/assets/owl-catalog.png new file mode 100644 index 00000000..426a6f44 Binary files /dev/null and b/.gitbook/assets/owl-catalog.png differ diff --git a/.gitbook/assets/owl-categorical-outlier.png b/.gitbook/assets/owl-categorical-outlier.png new file mode 100644 index 00000000..8f1e213c Binary files /dev/null and b/.gitbook/assets/owl-categorical-outlier.png differ diff --git a/.gitbook/assets/owl-cluster-health.png b/.gitbook/assets/owl-cluster-health.png new file mode 100644 index 00000000..a5843a45 Binary files /dev/null and b/.gitbook/assets/owl-cluster-health.png differ diff --git a/.gitbook/assets/owl-connection.png b/.gitbook/assets/owl-connection.png new file mode 100644 index 00000000..7e6e778d Binary files /dev/null and b/.gitbook/assets/owl-connection.png differ diff --git a/.gitbook/assets/owl-correlation.png b/.gitbook/assets/owl-correlation.png new file mode 100644 index 00000000..c710583a Binary files /dev/null and b/.gitbook/assets/owl-correlation.png differ diff --git a/.gitbook/assets/owl-databricks.png b/.gitbook/assets/owl-databricks.png new file mode 100644 index 00000000..6ab82e40 Binary files /dev/null and b/.gitbook/assets/owl-databricks.png differ diff --git a/.gitbook/assets/owl-db-explorer.png b/.gitbook/assets/owl-db-explorer.png new file mode 100644 index 00000000..e527eaf9 Binary files /dev/null and b/.gitbook/assets/owl-db-explorer.png differ diff --git a/.gitbook/assets/owl-df-with-hist-customer_id.png b/.gitbook/assets/owl-df-with-hist-customer_id.png new file mode 100644 index 00000000..0a562053 Binary files /dev/null and b/.gitbook/assets/owl-df-with-hist-customer_id.png differ diff --git a/.gitbook/assets/owl-dupe-booked.png b/.gitbook/assets/owl-dupe-booked.png new file mode 100644 index 00000000..c5438ff0 Binary files /dev/null and b/.gitbook/assets/owl-dupe-booked.png differ diff --git a/.gitbook/assets/owl-dupe-carrissa.png b/.gitbook/assets/owl-dupe-carrissa.png new file mode 100644 index 00000000..1eb41044 Binary files /dev/null and b/.gitbook/assets/owl-dupe-carrissa.png differ diff --git a/.gitbook/assets/owl-dupe-df.png b/.gitbook/assets/owl-dupe-df.png new file mode 100644 index 00000000..a76fdb76 Binary files /dev/null and b/.gitbook/assets/owl-dupe-df.png differ diff --git a/.gitbook/assets/owl-dupes.png b/.gitbook/assets/owl-dupes.png new file mode 100644 index 00000000..613880c4 Binary files /dev/null and b/.gitbook/assets/owl-dupes.png differ diff --git a/.gitbook/assets/owl-email-alert.png b/.gitbook/assets/owl-email-alert.png new file mode 100644 index 00000000..5a5b9e14 Binary files /dev/null and b/.gitbook/assets/owl-email-alert.png differ diff --git a/.gitbook/assets/owl-explorer (1).png b/.gitbook/assets/owl-explorer (1).png new file mode 100644 index 00000000..f100a04f Binary files /dev/null and b/.gitbook/assets/owl-explorer (1).png differ diff --git a/.gitbook/assets/owl-explorer.png b/.gitbook/assets/owl-explorer.png new file mode 100644 index 00000000..2771dcc3 Binary files /dev/null and b/.gitbook/assets/owl-explorer.png differ diff --git a/.gitbook/assets/owl-framework2.png b/.gitbook/assets/owl-framework2.png new file mode 100644 index 00000000..3cc8f9b7 Binary files /dev/null and b/.gitbook/assets/owl-framework2.png differ diff --git a/.gitbook/assets/owl-graph-insider-trading.png b/.gitbook/assets/owl-graph-insider-trading.png new file mode 100644 index 00000000..439384f1 Binary files /dev/null and b/.gitbook/assets/owl-graph-insider-trading.png differ diff --git a/.gitbook/assets/owl-group-scorecard.png b/.gitbook/assets/owl-group-scorecard.png new file mode 100644 index 00000000..0c0977cc Binary files /dev/null and b/.gitbook/assets/owl-group-scorecard.png differ diff --git a/.gitbook/assets/owl-health-chart.png b/.gitbook/assets/owl-health-chart.png new file mode 100644 index 00000000..2b3167e4 Binary files /dev/null and b/.gitbook/assets/owl-health-chart.png differ diff --git a/.gitbook/assets/owl-histogram.png b/.gitbook/assets/owl-histogram.png new file mode 100644 index 00000000..459f3db5 Binary files /dev/null and b/.gitbook/assets/owl-histogram.png differ diff --git a/.gitbook/assets/owl-item-labeling.png b/.gitbook/assets/owl-item-labeling.png new file mode 100644 index 00000000..84bafeeb Binary files /dev/null and b/.gitbook/assets/owl-item-labeling.png differ diff --git a/.gitbook/assets/owl-job-db.png b/.gitbook/assets/owl-job-db.png new file mode 100644 index 00000000..3847782a Binary files /dev/null and b/.gitbook/assets/owl-job-db.png differ diff --git a/.gitbook/assets/owl-job-status (1).png b/.gitbook/assets/owl-job-status (1).png new file mode 100644 index 00000000..64c70222 Binary files /dev/null and b/.gitbook/assets/owl-job-status (1).png differ diff --git a/.gitbook/assets/owl-job-status.png b/.gitbook/assets/owl-job-status.png new file mode 100644 index 00000000..64c70222 Binary files /dev/null and b/.gitbook/assets/owl-job-status.png differ diff --git a/.gitbook/assets/owl-k8s-deployment.png b/.gitbook/assets/owl-k8s-deployment.png new file mode 100644 index 00000000..5921acd7 Binary files /dev/null and b/.gitbook/assets/owl-k8s-deployment.png differ diff --git a/.gitbook/assets/owl-list-view.png b/.gitbook/assets/owl-list-view.png new file mode 100644 index 00000000..45bbdc31 Binary files /dev/null and b/.gitbook/assets/owl-list-view.png differ diff --git a/.gitbook/assets/owl-logo.png b/.gitbook/assets/owl-logo.png new file mode 100644 index 00000000..0e267201 Binary files /dev/null and b/.gitbook/assets/owl-logo.png differ diff --git a/.gitbook/assets/owl-missing-records.png b/.gitbook/assets/owl-missing-records.png new file mode 100644 index 00000000..6afe1a8a Binary files /dev/null and b/.gitbook/assets/owl-missing-records.png differ diff --git a/.gitbook/assets/owl-mysql.png b/.gitbook/assets/owl-mysql.png new file mode 100644 index 00000000..b2830f16 Binary files /dev/null and b/.gitbook/assets/owl-mysql.png differ diff --git a/.gitbook/assets/owl-off-peak.png b/.gitbook/assets/owl-off-peak.png new file mode 100644 index 00000000..12818363 Binary files /dev/null and b/.gitbook/assets/owl-off-peak.png differ diff --git a/.gitbook/assets/owl-outlier-numerical (1).png b/.gitbook/assets/owl-outlier-numerical (1).png new file mode 100644 index 00000000..dba87759 Binary files /dev/null and b/.gitbook/assets/owl-outlier-numerical (1).png differ diff --git a/.gitbook/assets/owl-outlier-numerical (2).png b/.gitbook/assets/owl-outlier-numerical (2).png new file mode 100644 index 00000000..9932a21c Binary files /dev/null and b/.gitbook/assets/owl-outlier-numerical (2).png differ diff --git a/.gitbook/assets/owl-outlier-numerical.png b/.gitbook/assets/owl-outlier-numerical.png new file mode 100644 index 00000000..dba87759 Binary files /dev/null and b/.gitbook/assets/owl-outlier-numerical.png differ diff --git a/.gitbook/assets/owl-outliers.png b/.gitbook/assets/owl-outliers.png new file mode 100644 index 00000000..4a24a689 Binary files /dev/null and b/.gitbook/assets/owl-outliers.png differ diff --git a/.gitbook/assets/owl-parallel-jdbc.png b/.gitbook/assets/owl-parallel-jdbc.png new file mode 100644 index 00000000..ee2725e8 Binary files /dev/null and b/.gitbook/assets/owl-parallel-jdbc.png differ diff --git a/.gitbook/assets/owl-patterns.png b/.gitbook/assets/owl-patterns.png new file mode 100644 index 00000000..e1c8f43a Binary files /dev/null and b/.gitbook/assets/owl-patterns.png differ diff --git a/.gitbook/assets/owl-phone-shapes.png b/.gitbook/assets/owl-phone-shapes.png new file mode 100644 index 00000000..86e45100 Binary files /dev/null and b/.gitbook/assets/owl-phone-shapes.png differ diff --git a/.gitbook/assets/owl-pii.png b/.gitbook/assets/owl-pii.png new file mode 100644 index 00000000..6c0d6b45 Binary files /dev/null and b/.gitbook/assets/owl-pii.png differ diff --git a/.gitbook/assets/owl-pipeline.png b/.gitbook/assets/owl-pipeline.png new file mode 100644 index 00000000..bde95e30 Binary files /dev/null and b/.gitbook/assets/owl-pipeline.png differ diff --git a/.gitbook/assets/owl-profile (1).png b/.gitbook/assets/owl-profile (1).png new file mode 100644 index 00000000..7d986fcc Binary files /dev/null and b/.gitbook/assets/owl-profile (1).png differ diff --git a/.gitbook/assets/owl-profile-delta.png b/.gitbook/assets/owl-profile-delta.png new file mode 100644 index 00000000..1761e181 Binary files /dev/null and b/.gitbook/assets/owl-profile-delta.png differ diff --git a/.gitbook/assets/owl-profile.png b/.gitbook/assets/owl-profile.png new file mode 100644 index 00000000..9dba82c7 Binary files /dev/null and b/.gitbook/assets/owl-profile.png differ diff --git a/.gitbook/assets/owl-profiler.png b/.gitbook/assets/owl-profiler.png new file mode 100644 index 00000000..f2740213 Binary files /dev/null and b/.gitbook/assets/owl-profiler.png differ diff --git a/.gitbook/assets/owl-relationships.png b/.gitbook/assets/owl-relationships.png new file mode 100644 index 00000000..1dd0d426 Binary files /dev/null and b/.gitbook/assets/owl-relationships.png differ diff --git a/.gitbook/assets/owl-rich-labeling.png b/.gitbook/assets/owl-rich-labeling.png new file mode 100644 index 00000000..6fe542ec Binary files /dev/null and b/.gitbook/assets/owl-rich-labeling.png differ diff --git a/.gitbook/assets/owl-row-trend.png b/.gitbook/assets/owl-row-trend.png new file mode 100644 index 00000000..6da02ffe Binary files /dev/null and b/.gitbook/assets/owl-row-trend.png differ diff --git a/.gitbook/assets/owl-rule-builder-custom.png b/.gitbook/assets/owl-rule-builder-custom.png new file mode 100644 index 00000000..31b2c73c Binary files /dev/null and b/.gitbook/assets/owl-rule-builder-custom.png differ diff --git a/.gitbook/assets/owl-rule-builder-zoomed (1).png b/.gitbook/assets/owl-rule-builder-zoomed (1).png new file mode 100644 index 00000000..3dbb0af8 Binary files /dev/null and b/.gitbook/assets/owl-rule-builder-zoomed (1).png differ diff --git a/.gitbook/assets/owl-rule-builder-zoomed.png b/.gitbook/assets/owl-rule-builder-zoomed.png new file mode 100644 index 00000000..a57b0ddb Binary files /dev/null and b/.gitbook/assets/owl-rule-builder-zoomed.png differ diff --git a/.gitbook/assets/owl-rule-builder.png b/.gitbook/assets/owl-rule-builder.png new file mode 100644 index 00000000..0010a9c3 Binary files /dev/null and b/.gitbook/assets/owl-rule-builder.png differ diff --git a/.gitbook/assets/owl-rule-repo (1).png b/.gitbook/assets/owl-rule-repo (1).png new file mode 100644 index 00000000..4cf3f640 Binary files /dev/null and b/.gitbook/assets/owl-rule-repo (1).png differ diff --git a/.gitbook/assets/owl-rule-repo (2).png b/.gitbook/assets/owl-rule-repo (2).png new file mode 100644 index 00000000..4cf3f640 Binary files /dev/null and b/.gitbook/assets/owl-rule-repo (2).png differ diff --git a/.gitbook/assets/owl-rule-repo (3).png b/.gitbook/assets/owl-rule-repo (3).png new file mode 100644 index 00000000..4cf3f640 Binary files /dev/null and b/.gitbook/assets/owl-rule-repo (3).png differ diff --git a/.gitbook/assets/owl-rule-repo.png b/.gitbook/assets/owl-rule-repo.png new file mode 100644 index 00000000..4cf3f640 Binary files /dev/null and b/.gitbook/assets/owl-rule-repo.png differ diff --git a/.gitbook/assets/owl-schedule.png b/.gitbook/assets/owl-schedule.png new file mode 100644 index 00000000..d1b086ee Binary files /dev/null and b/.gitbook/assets/owl-schedule.png differ diff --git a/.gitbook/assets/owl-scheduler (1).png b/.gitbook/assets/owl-scheduler (1).png new file mode 100644 index 00000000..65dd776e Binary files /dev/null and b/.gitbook/assets/owl-scheduler (1).png differ diff --git a/.gitbook/assets/owl-scheduler.png b/.gitbook/assets/owl-scheduler.png new file mode 100644 index 00000000..65dd776e Binary files /dev/null and b/.gitbook/assets/owl-scheduler.png differ diff --git a/.gitbook/assets/owl-schema.png b/.gitbook/assets/owl-schema.png new file mode 100644 index 00000000..4257e148 Binary files /dev/null and b/.gitbook/assets/owl-schema.png differ diff --git a/.gitbook/assets/owl-scorecard.png b/.gitbook/assets/owl-scorecard.png new file mode 100644 index 00000000..77e0d87d Binary files /dev/null and b/.gitbook/assets/owl-scorecard.png differ diff --git a/.gitbook/assets/owl-sensor-streams.png b/.gitbook/assets/owl-sensor-streams.png new file mode 100644 index 00000000..ad53141c Binary files /dev/null and b/.gitbook/assets/owl-sensor-streams.png differ diff --git a/.gitbook/assets/owl-shape-drillin.png b/.gitbook/assets/owl-shape-drillin.png new file mode 100644 index 00000000..618247b2 Binary files /dev/null and b/.gitbook/assets/owl-shape-drillin.png differ diff --git a/.gitbook/assets/owl-spark-dq=pipeline.png b/.gitbook/assets/owl-spark-dq=pipeline.png new file mode 100644 index 00000000..1a521c1d Binary files /dev/null and b/.gitbook/assets/owl-spark-dq=pipeline.png differ diff --git a/.gitbook/assets/owl-sql-editor.png b/.gitbook/assets/owl-sql-editor.png new file mode 100644 index 00000000..e60b771f Binary files /dev/null and b/.gitbook/assets/owl-sql-editor.png differ diff --git a/.gitbook/assets/owl-stock-eod-graph.png b/.gitbook/assets/owl-stock-eod-graph.png new file mode 100644 index 00000000..441154e6 Binary files /dev/null and b/.gitbook/assets/owl-stock-eod-graph.png differ diff --git a/.gitbook/assets/owl-suite.png b/.gitbook/assets/owl-suite.png new file mode 100644 index 00000000..626ea5e5 Binary files /dev/null and b/.gitbook/assets/owl-suite.png differ diff --git a/.gitbook/assets/owl-swagger.png b/.gitbook/assets/owl-swagger.png new file mode 100644 index 00000000..34df2088 Binary files /dev/null and b/.gitbook/assets/owl-swagger.png differ diff --git a/.gitbook/assets/owl-template-schedule.png b/.gitbook/assets/owl-template-schedule.png new file mode 100644 index 00000000..537e7703 Binary files /dev/null and b/.gitbook/assets/owl-template-schedule.png differ diff --git a/.gitbook/assets/owl-tenant-agent.png b/.gitbook/assets/owl-tenant-agent.png new file mode 100644 index 00000000..ce597a10 Binary files /dev/null and b/.gitbook/assets/owl-tenant-agent.png differ diff --git a/.gitbook/assets/owl-trend.png b/.gitbook/assets/owl-trend.png new file mode 100644 index 00000000..f3c924d7 Binary files /dev/null and b/.gitbook/assets/owl-trend.png differ diff --git a/.gitbook/assets/owl-unified-dq.jpg b/.gitbook/assets/owl-unified-dq.jpg new file mode 100644 index 00000000..c14709e2 Binary files /dev/null and b/.gitbook/assets/owl-unified-dq.jpg differ diff --git a/.gitbook/assets/owl-zeppelin-timeout.png b/.gitbook/assets/owl-zeppelin-timeout.png new file mode 100644 index 00000000..1499f128 Binary files /dev/null and b/.gitbook/assets/owl-zeppelin-timeout.png differ diff --git a/.gitbook/assets/owldq-framework-li.png b/.gitbook/assets/owldq-framework-li.png new file mode 100644 index 00000000..8fd1ee05 Binary files /dev/null and b/.gitbook/assets/owldq-framework-li.png differ diff --git a/.gitbook/assets/owldq-framework-pipeline.png b/.gitbook/assets/owldq-framework-pipeline.png new file mode 100644 index 00000000..4d4c6efa Binary files /dev/null and b/.gitbook/assets/owldq-framework-pipeline.png differ diff --git a/.gitbook/assets/owldq-framework.png b/.gitbook/assets/owldq-framework.png new file mode 100644 index 00000000..b5cdda59 Binary files /dev/null and b/.gitbook/assets/owldq-framework.png differ diff --git a/.gitbook/assets/owldq-framework2.png b/.gitbook/assets/owldq-framework2.png new file mode 100644 index 00000000..cc07c7ae Binary files /dev/null and b/.gitbook/assets/owldq-framework2.png differ diff --git a/.gitbook/assets/owldq-healthcare (1).jpg b/.gitbook/assets/owldq-healthcare (1).jpg new file mode 100644 index 00000000..1c88a8ca Binary files /dev/null and b/.gitbook/assets/owldq-healthcare (1).jpg differ diff --git a/.gitbook/assets/owlimport.mp4 b/.gitbook/assets/owlimport.mp4 new file mode 100644 index 00000000..583c1630 Binary files /dev/null and b/.gitbook/assets/owlimport.mp4 differ diff --git a/.gitbook/assets/page_view.gif b/.gitbook/assets/page_view.gif new file mode 100644 index 00000000..03a1b72f Binary files /dev/null and b/.gitbook/assets/page_view.gif differ diff --git a/.gitbook/assets/positions-owl (1).png b/.gitbook/assets/positions-owl (1).png new file mode 100644 index 00000000..3c025880 Binary files /dev/null and b/.gitbook/assets/positions-owl (1).png differ diff --git a/.gitbook/assets/positions-owl.png b/.gitbook/assets/positions-owl.png new file mode 100644 index 00000000..3c025880 Binary files /dev/null and b/.gitbook/assets/positions-owl.png differ diff --git a/.gitbook/assets/profile_options (1).gif b/.gitbook/assets/profile_options (1).gif new file mode 100644 index 00000000..955f5696 Binary files /dev/null and b/.gitbook/assets/profile_options (1).gif differ diff --git a/.gitbook/assets/profile_options.gif b/.gitbook/assets/profile_options.gif new file mode 100644 index 00000000..955f5696 Binary files /dev/null and b/.gitbook/assets/profile_options.gif differ diff --git a/.gitbook/assets/profile_results (1).gif b/.gitbook/assets/profile_results (1).gif new file mode 100644 index 00000000..e6153774 Binary files /dev/null and b/.gitbook/assets/profile_results (1).gif differ diff --git a/.gitbook/assets/profile_results.gif b/.gitbook/assets/profile_results.gif new file mode 100644 index 00000000..e6153774 Binary files /dev/null and b/.gitbook/assets/profile_results.gif differ diff --git a/.gitbook/assets/profile_scope.gif b/.gitbook/assets/profile_scope.gif new file mode 100644 index 00000000..e0fbaf73 Binary files /dev/null and b/.gitbook/assets/profile_scope.gif differ diff --git a/.gitbook/assets/profile_setup (1).gif b/.gitbook/assets/profile_setup (1).gif new file mode 100644 index 00000000..eea38202 Binary files /dev/null and b/.gitbook/assets/profile_setup (1).gif differ diff --git a/.gitbook/assets/profile_setup.gif b/.gitbook/assets/profile_setup.gif new file mode 100644 index 00000000..ff4c068d Binary files /dev/null and b/.gitbook/assets/profile_setup.gif differ diff --git a/.gitbook/assets/pushdown (1).gif b/.gitbook/assets/pushdown (1).gif new file mode 100644 index 00000000..ed927d5d Binary files /dev/null and b/.gitbook/assets/pushdown (1).gif differ diff --git a/.gitbook/assets/pushdown.gif b/.gitbook/assets/pushdown.gif new file mode 100644 index 00000000..ed927d5d Binary files /dev/null and b/.gitbook/assets/pushdown.gif differ diff --git a/.gitbook/assets/query_builder.gif b/.gitbook/assets/query_builder.gif new file mode 100644 index 00000000..6a28c489 Binary files /dev/null and b/.gitbook/assets/query_builder.gif differ diff --git a/.gitbook/assets/quick_rules (1).gif b/.gitbook/assets/quick_rules (1).gif new file mode 100644 index 00000000..7fd67e74 Binary files /dev/null and b/.gitbook/assets/quick_rules (1).gif differ diff --git a/.gitbook/assets/quick_rules.gif b/.gitbook/assets/quick_rules.gif new file mode 100644 index 00000000..7fd67e74 Binary files /dev/null and b/.gitbook/assets/quick_rules.gif differ diff --git a/.gitbook/assets/reports (1).gif b/.gitbook/assets/reports (1).gif new file mode 100644 index 00000000..e943460a Binary files /dev/null and b/.gitbook/assets/reports (1).gif differ diff --git a/.gitbook/assets/reports.gif b/.gitbook/assets/reports.gif new file mode 100644 index 00000000..e943460a Binary files /dev/null and b/.gitbook/assets/reports.gif differ diff --git a/.gitbook/assets/review-dataconcept-results.gif b/.gitbook/assets/review-dataconcept-results.gif new file mode 100644 index 00000000..3f8e77c0 Binary files /dev/null and b/.gitbook/assets/review-dataconcept-results.gif differ diff --git a/.gitbook/assets/rule_.gif b/.gitbook/assets/rule_.gif new file mode 100644 index 00000000..26193bca Binary files /dev/null and b/.gitbook/assets/rule_.gif differ diff --git a/.gitbook/assets/rule_dimensions.gif b/.gitbook/assets/rule_dimensions.gif new file mode 100644 index 00000000..8ce44905 Binary files /dev/null and b/.gitbook/assets/rule_dimensions.gif differ diff --git a/.gitbook/assets/rule_lib (1).gif b/.gitbook/assets/rule_lib (1).gif new file mode 100644 index 00000000..ea3d4d50 Binary files /dev/null and b/.gitbook/assets/rule_lib (1).gif differ diff --git a/.gitbook/assets/rule_lib.gif b/.gitbook/assets/rule_lib.gif new file mode 100644 index 00000000..ea3d4d50 Binary files /dev/null and b/.gitbook/assets/rule_lib.gif differ diff --git a/.gitbook/assets/rule_nav.gif b/.gitbook/assets/rule_nav.gif new file mode 100644 index 00000000..dedc8dc1 Binary files /dev/null and b/.gitbook/assets/rule_nav.gif differ diff --git a/.gitbook/assets/rule_scoring (1).gif b/.gitbook/assets/rule_scoring (1).gif new file mode 100644 index 00000000..f07e43ee Binary files /dev/null and b/.gitbook/assets/rule_scoring (1).gif differ diff --git a/.gitbook/assets/rule_scoring.gif b/.gitbook/assets/rule_scoring.gif new file mode 100644 index 00000000..f07e43ee Binary files /dev/null and b/.gitbook/assets/rule_scoring.gif differ diff --git a/.gitbook/assets/rule_template (1).gif b/.gitbook/assets/rule_template (1).gif new file mode 100644 index 00000000..e8c18613 Binary files /dev/null and b/.gitbook/assets/rule_template (1).gif differ diff --git a/.gitbook/assets/rule_template.gif b/.gitbook/assets/rule_template.gif new file mode 100644 index 00000000..e8c18613 Binary files /dev/null and b/.gitbook/assets/rule_template.gif differ diff --git a/.gitbook/assets/rules (1).gif b/.gitbook/assets/rules (1).gif new file mode 100644 index 00000000..d0a737f0 Binary files /dev/null and b/.gitbook/assets/rules (1).gif differ diff --git a/.gitbook/assets/rules.gif b/.gitbook/assets/rules.gif new file mode 100644 index 00000000..5a96038d Binary files /dev/null and b/.gitbook/assets/rules.gif differ diff --git a/.gitbook/assets/sample_Infosec_Data.png b/.gitbook/assets/sample_Infosec_Data.png new file mode 100644 index 00000000..21179196 Binary files /dev/null and b/.gitbook/assets/sample_Infosec_Data.png differ diff --git a/.gitbook/assets/schedule.gif b/.gitbook/assets/schedule.gif new file mode 100644 index 00000000..6aac5dde Binary files /dev/null and b/.gitbook/assets/schedule.gif differ diff --git a/.gitbook/assets/scorecards (1).gif b/.gitbook/assets/scorecards (1).gif new file mode 100644 index 00000000..ca605ba8 Binary files /dev/null and b/.gitbook/assets/scorecards (1).gif differ diff --git a/.gitbook/assets/scorecards (2).gif b/.gitbook/assets/scorecards (2).gif new file mode 100644 index 00000000..f3c5fa7b Binary files /dev/null and b/.gitbook/assets/scorecards (2).gif differ diff --git a/.gitbook/assets/scorecards.gif b/.gitbook/assets/scorecards.gif new file mode 100644 index 00000000..20a10b35 Binary files /dev/null and b/.gitbook/assets/scorecards.gif differ diff --git a/.gitbook/assets/scorecards2.gif b/.gitbook/assets/scorecards2.gif new file mode 100644 index 00000000..7701a211 Binary files /dev/null and b/.gitbook/assets/scorecards2.gif differ diff --git a/.gitbook/assets/scoring.gif b/.gitbook/assets/scoring.gif new file mode 100644 index 00000000..39dc944b Binary files /dev/null and b/.gitbook/assets/scoring.gif differ diff --git a/.gitbook/assets/screen-shot-2019-07-17-at-10.55.06-am (1).png b/.gitbook/assets/screen-shot-2019-07-17-at-10.55.06-am (1).png new file mode 100644 index 00000000..efb05ea6 Binary files /dev/null and b/.gitbook/assets/screen-shot-2019-07-17-at-10.55.06-am (1).png differ diff --git a/.gitbook/assets/screen-shot-2019-09-03-at-11.13.27-am.png b/.gitbook/assets/screen-shot-2019-09-03-at-11.13.27-am.png new file mode 100644 index 00000000..3db5d52e Binary files /dev/null and b/.gitbook/assets/screen-shot-2019-09-03-at-11.13.27-am.png differ diff --git a/.gitbook/assets/screen-shot-2019-11-04-at-2.02.12-pm (1).png b/.gitbook/assets/screen-shot-2019-11-04-at-2.02.12-pm (1).png new file mode 100644 index 00000000..77482255 Binary files /dev/null and b/.gitbook/assets/screen-shot-2019-11-04-at-2.02.12-pm (1).png differ diff --git a/.gitbook/assets/screen-shot-2019-12-18-at-1.50.34-pm.png b/.gitbook/assets/screen-shot-2019-12-18-at-1.50.34-pm.png new file mode 100644 index 00000000..9135e327 Binary files /dev/null and b/.gitbook/assets/screen-shot-2019-12-18-at-1.50.34-pm.png differ diff --git a/.gitbook/assets/screen-shot-2020-05-07-at-6.55.18-pm (1).png b/.gitbook/assets/screen-shot-2020-05-07-at-6.55.18-pm (1).png new file mode 100644 index 00000000..0f4e56fb Binary files /dev/null and b/.gitbook/assets/screen-shot-2020-05-07-at-6.55.18-pm (1).png differ diff --git a/.gitbook/assets/screen-shot-2020-07-07-at-5.06.59-am (1).png b/.gitbook/assets/screen-shot-2020-07-07-at-5.06.59-am (1).png new file mode 100644 index 00000000..96d4b4d3 Binary files /dev/null and b/.gitbook/assets/screen-shot-2020-07-07-at-5.06.59-am (1).png differ diff --git a/.gitbook/assets/screen-shot-2020-07-07-at-5.27.41-am (2) (1).png b/.gitbook/assets/screen-shot-2020-07-07-at-5.27.41-am (2) (1).png new file mode 100644 index 00000000..d9b63636 Binary files /dev/null and b/.gitbook/assets/screen-shot-2020-07-07-at-5.27.41-am (2) (1).png differ diff --git a/.gitbook/assets/screen-shot-2020-07-07-at-5.27.41-am (2).png b/.gitbook/assets/screen-shot-2020-07-07-at-5.27.41-am (2).png new file mode 100644 index 00000000..d9b63636 Binary files /dev/null and b/.gitbook/assets/screen-shot-2020-07-07-at-5.27.41-am (2).png differ diff --git a/.gitbook/assets/screen-shot-2021-04-27-at-8.07.58-am (1).png b/.gitbook/assets/screen-shot-2021-04-27-at-8.07.58-am (1).png new file mode 100644 index 00000000..f9873917 Binary files /dev/null and b/.gitbook/assets/screen-shot-2021-04-27-at-8.07.58-am (1).png differ diff --git a/.gitbook/assets/screenshot-2020-07-18-at-1.47.01-pm (3).png b/.gitbook/assets/screenshot-2020-07-18-at-1.47.01-pm (3).png new file mode 100644 index 00000000..c9416bda Binary files /dev/null and b/.gitbook/assets/screenshot-2020-07-18-at-1.47.01-pm (3).png differ diff --git a/.gitbook/assets/screenshot-2020-07-18-at-1.47.01-pm (4).png b/.gitbook/assets/screenshot-2020-07-18-at-1.47.01-pm (4).png new file mode 100644 index 00000000..7301c9e6 Binary files /dev/null and b/.gitbook/assets/screenshot-2020-07-18-at-1.47.01-pm (4).png differ diff --git a/.gitbook/assets/screenshot.png b/.gitbook/assets/screenshot.png new file mode 100644 index 00000000..2d98247c Binary files /dev/null and b/.gitbook/assets/screenshot.png differ diff --git a/.gitbook/assets/semantic-apply-dataconcept (1).gif b/.gitbook/assets/semantic-apply-dataconcept (1).gif new file mode 100644 index 00000000..fbafeab5 Binary files /dev/null and b/.gitbook/assets/semantic-apply-dataconcept (1).gif differ diff --git a/.gitbook/assets/semantic-apply-dataconcept.gif b/.gitbook/assets/semantic-apply-dataconcept.gif new file mode 100644 index 00000000..594a456d Binary files /dev/null and b/.gitbook/assets/semantic-apply-dataconcept.gif differ diff --git a/.gitbook/assets/shape-tuning-owl.png b/.gitbook/assets/shape-tuning-owl.png new file mode 100644 index 00000000..fb9d3744 Binary files /dev/null and b/.gitbook/assets/shape-tuning-owl.png differ diff --git a/.gitbook/assets/shapes (1).jpg b/.gitbook/assets/shapes (1).jpg new file mode 100644 index 00000000..1837109e Binary files /dev/null and b/.gitbook/assets/shapes (1).jpg differ diff --git a/.gitbook/assets/smtp_setup.gif b/.gitbook/assets/smtp_setup.gif new file mode 100644 index 00000000..298a0e45 Binary files /dev/null and b/.gitbook/assets/smtp_setup.gif differ diff --git a/.gitbook/assets/source.jpg b/.gitbook/assets/source.jpg new file mode 100644 index 00000000..a017923b Binary files /dev/null and b/.gitbook/assets/source.jpg differ diff --git a/.gitbook/assets/source.png b/.gitbook/assets/source.png new file mode 100644 index 00000000..6dba959f Binary files /dev/null and b/.gitbook/assets/source.png differ diff --git a/.gitbook/assets/sql_functions (1).gif b/.gitbook/assets/sql_functions (1).gif new file mode 100644 index 00000000..ac209f73 Binary files /dev/null and b/.gitbook/assets/sql_functions (1).gif differ diff --git a/.gitbook/assets/sql_functions.gif b/.gitbook/assets/sql_functions.gif new file mode 100644 index 00000000..ac209f73 Binary files /dev/null and b/.gitbook/assets/sql_functions.gif differ diff --git a/.gitbook/assets/swagger.gif b/.gitbook/assets/swagger.gif new file mode 100644 index 00000000..33938167 Binary files /dev/null and b/.gitbook/assets/swagger.gif differ diff --git a/.gitbook/assets/table-params.png b/.gitbook/assets/table-params.png new file mode 100644 index 00000000..0945fe1d Binary files /dev/null and b/.gitbook/assets/table-params.png differ diff --git a/README.md b/README.md new file mode 100644 index 00000000..157f593f --- /dev/null +++ b/README.md @@ -0,0 +1,39 @@ +--- +description: Autogenerate Rules, Anomaly Detection, Reconciliation +--- + +# Collibra Data Quality & Observability + +![](<.gitbook/assets/Screen Shot 2021-07-28 at 8.02.40 AM.png>) + +Automatic Data Quality without the need for Rules. Owl provides a fast and elegant way to manage your data sets by learning through observation rather than human input. Owl applies the latest advancements in Data Science and Machine Learning to the problem of Data Quality. Surfacing data issues in minutes instead of months. + +![](<.gitbook/assets/Statistical Process Control for Data Quality (Animation) Aug 2021.gif>) + +## Getting Started with Collibra DQ + +{% embed url="https://vimeo.com/372286610" %} + + + +## Deep Learning vs Machine Learning + +{% embed url="https://vimeo.com/312445160" %} + +## What Does DQ Mean to You? + +{% embed url="https://www.youtube.com/watch?v=6I343v1lmDM" %} + + + +### A Pluggable and Complete Data Quality Framework + +If you are adding data quality to your data pipelines the below visual illustrates the number of products and pieces you will need consider to successfully complete your overall governance program. The OwlDQ suite allows you to use either native Owl components or integrate the 3rd party components of your choice. By using our best practice guide and framework you can complete the DQ lifecycle easily. + +![](.gitbook/assets/owldq-framework.png) + +## Data Pipelines that Tie into the DQ Framework + +Owl offers a coding framework for developers or ETL designers that want to built real-time data quality into their broader data pipeline. This provides the same algos and DQ checks as the Owl UI Wizard but with direct access into your code points. Consistency is a must to have a program you trust. + +![](.gitbook/assets/owldq-framework-pipeline.png) diff --git a/SUMMARY.md b/SUMMARY.md new file mode 100644 index 00000000..0ff18cac --- /dev/null +++ b/SUMMARY.md @@ -0,0 +1,251 @@ +# Table of contents + +* [Collibra Data Quality & Observability](README.md) +* [Release Notes](release-notes.md) + +## Installation + +* [Standalone](installation/standalone/README.md) + * [Standalone Install](installation/standalone/full-install.md) + * [Standalone Upgrade](installation/standalone/standalone-upgrade.md) + * [Standalone Sizing](installation/standalone/standalone-sizing.md) +* [Hadoop](installation/hadoop/README.md) + * [Hadoop Install](installation/hadoop/hadoop-integration.md) + * [EMR / Dataproc / HDI](installation/hadoop/emr.md) +* [Cloud native](installation/cloud-native-owldq/README.md) + * [Cloud native requirements](installation/cloud-native-owldq/preparing-for-deployment.md) + * [Cloud native install](installation/cloud-native-owldq/deploying-cloud-native-owldq.md) + * [EKS / GKE / AKS](installation/cloud-native-owldq/aks-eks-gke-kubernetes-deployment.md) +* [Agent](installation/agent-configuration.md) + +## Connections + +* [Supported Connections](connecting-to-dbs-in-owl-web/supported-drivers/README.md) + * [Connectivity to Athena](connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-athena.md) + * [Connectivity to BigQuery](connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-bigquery.md) + * [Connectivity to Hive](connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-hive/README.md) + * [Connecting to CDH 5.16 Hive SSL/TLS/Kerberos Setup](connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-hive/connecting-to-cdh-5.16-hive-ssl-tls-kerberos-setup.md) + * [Connectivity to Oracle](connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-oracle.md) + * [Connectivity to Presto](connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-presto.md) + * [Connectivity to Redshift](connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-redshift.md) + * [Connectivity to Snowflake](connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-snowflake.md) + * [Connectivity to SQL Server](connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-sql-server.md) +* [Add a Connection](connecting-to-dbs-in-owl-web/owl-db-connection.md) +* [Add Connection to Agent](connecting-to-dbs-in-owl-web/add-connection-to-agent.md) + +## Features + +* [Profile (automatic)](dq-visuals/profile.md) +* [Behavior (automatic)](dq-visuals/behaviors.md) +* [Rules (user-defined)](dq-visuals/rules/README.md) + * [Adding a Rule](dq-visuals/rules/creating-a-business-rule.md) + * [Scores](dq-visuals/rules/scores.md) + * [Dimensions](dq-visuals/rules/dimensions.md) + * [Rule Types](dq-visuals/rules/rule-types/README.md) + * [SQL Based Rules](dq-visuals/rules/rule-types/sql-based-rules/README.md) + * [Simple](dq-visuals/rules/rule-types/sql-based-rules/simple.md) + * [Freeform](dq-visuals/rules/rule-types/sql-based-rules/freeform/README.md) + * [Cross-Dataset Rules](dq-visuals/rules/rule-types/sql-based-rules/freeform/easier-rule-examples.md) + * [Native](dq-visuals/rules/rule-types/sql-based-rules/native-sql.md) + * [Stat Rules](dq-visuals/rules/rule-types/stat-rules/README.md) + * [Data Type Rules](dq-visuals/rules/rule-types/stat-rules/data-type-rules.md) + * [Rule Templates](dq-visuals/rules/rule-templates.md) + * [Rule Library](dq-visuals/rules/dq-rule-automation.md) + * [Rule Discovery](dq-visuals/rules/data-concepts-and-semantics.md) +* [More](dq-visuals/more/README.md) + * [Explorer (no-code)](dq-visuals/more/explorer-2.md) + * [Schema (automatic)](dq-visuals/more/schema-evolution.md) + * [Shapes (automatic)](dq-visuals/more/shapes.md) + * [Duplicates (advanced)](dq-visuals/more/duplicates.md) + * [Outliers (advanced)](dq-visuals/more/outliers.md) + * [Patterns (advanced)](dq-visuals/more/pattern-mining.md) + * [Records (advanced)](dq-visuals/more/missing-records.md) + * [Source (advanced)](dq-visuals/more/validate-source.md) + * [Pushdown](dq-visuals/more/pushdown.md) + * [Summary](dq-visuals/more/overview.md) + +## DQ JOB Examples + +* [Batch](dq-job-examples/owlcheck-spark/README.md) + * [DQ Job JDBC](dq-job-examples/owlcheck-spark/job-jdbc.md) + * [DQ Job BigQuery](dq-job-examples/owlcheck-spark/owlcheck-bigquery.md) + * [DQ Job Databricks](dq-job-examples/owlcheck-spark/owlcheck-databricks.md) + * [DQ Job Hive](dq-job-examples/owlcheck-spark/job-hive.md) + * [DQ Job Files](dq-job-examples/owlcheck-spark/job-files.md) + * [DQ Job HDFS](dq-job-examples/owlcheck-spark/owlcheck-hdfs.md) + * [DQ Job JSON](dq-job-examples/owlcheck-spark/job-json.md) + * [DQ Job MySql](dq-job-examples/owlcheck-spark/job-mysql.md) + * [DQ Job MongoDB](dq-job-examples/owlcheck-spark/owlcheck-mongodb.md) + * [DQ Job S3](dq-job-examples/owlcheck-spark/owlcheck-s3.md) + * [DQ Job Snowflake](dq-job-examples/owlcheck-spark/owlcheck-snowflake.md) +* [Pipelines](dq-job-examples/data-quality-pipelines/README.md) + * [Simple](dq-job-examples/data-quality-pipelines/simple-spark-data-pipeline.md) + * [Advanced](dq-job-examples/data-quality-pipelines/spark-dq-pipeline.md) + * [Rules](dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/README.md) + * [Global rules](dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/global-rules.md) + * [SQL based rules](dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/README.md) + * [Simple rule](dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/simple-rule.md) + * [Freeform SQL](dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/freeform-sql.md) + * [Data type based rules](dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/data-type-based-rules.md) + * [Outliers](dq-job-examples/data-quality-pipelines/notebook-outlier-example.md) + * [Column Match](dq-job-examples/data-quality-pipelines/notebook-colmatch-example.md) + * [AWS DataBricks](dq-job-examples/data-quality-pipelines/aws-databricks-dq-pipeline.md) + * [Azure DataBricks](dq-job-examples/data-quality-pipelines/azure-databricks-dq-pipeline.md) + * [Options API](dq-job-examples/data-quality-pipelines/owl-options-api.md) + * [FAQs](dq-job-examples/data-quality-pipelines/frequently-asked-questions.md) +* [Advanced](dq-job-examples/owlcheck/README.md) + * [DQ Job Back Run](dq-job-examples/owlcheck/owlcheck-backrun.md) + * [DQ Job Cron](dq-job-examples/owlcheck/owlcheck-cron.md) + * [DQ Job Kafka](dq-job-examples/owlcheck/job-kafka.md) + * [DQ Job LinkId](dq-job-examples/owlcheck/owlcheck-linkid.md) + * [DQ Job Validate Source](dq-job-examples/owlcheck/owlcheck-validate-source.md) + * [DQ Job 43M rows](dq-job-examples/owlcheck/owlcheck-43m-rows.md) + * [Add Date Column](dq-job-examples/owlcheck/add-date-column.md) + * [AutoProfile](dq-job-examples/owlcheck/autoprofile.md) + * [Cloudera CLASSPATH](dq-job-examples/owlcheck/cloudera-classpath.md) + * [Column Matching](dq-job-examples/owlcheck/cluster-health-report.md) + * [Date Time Variable Options](dq-job-examples/owlcheck/date-time-variable-options.md) + * [Deploy Mode](dq-job-examples/owlcheck/deploy-mode.md) + * [Explorer (advanced)](dq-job-examples/owlcheck/explorer.md) + * [File Look Back](dq-job-examples/owlcheck/file-lookback.md) + * [Filter & Filter Not](dq-job-examples/owlcheck/filter.md) + * [Multiple Pattern Relationships](dq-job-examples/owlcheck/multiple-pattern-combinations.md) + * [Nulls in Datasets](dq-job-examples/owlcheck/zero-if-null.md) + * [Transform](dq-job-examples/owlcheck/owlcheck-transform.md) + +## Collibra DIC Integration + +* [DQ Connector](integration/dq-connector.md) +* [DQ Workflows](integration/dq-workflows-in-collibra-dic.md) + +## Scorecards + +* [Overview](scorecards/dataset-scorecard/README.md) + * [Scoring](scorecards/dataset-scorecard/owl-dq-screen-shots.md) +* [Page View](scorecards/group-scorecard.md) +* [List View](scorecards/list-view.md) + +## Scheduler + +* [Schedule a Job](scheduler/schedule-owlchecks.md) +* [Schedule Management](scheduler/schedule-management/README.md) + * [View/Re-Run Scheduled Jobs](scheduler/schedule-management/view-re-run-scheduled-jobs.md) + +## WORKFLOWS / ALERTS + +* [Email Alerts](workflows/email-alerts.md) +* [Assignment Queue(s)](workflows/assignment-queue-s/README.md) + * [Internal Assignment](workflows/assignment-queue-s/internal-assignment.md) + * [External assignment](workflows/assignment-queue-s/external-assignment.md) + * [FAQ](workflows/assignment-queue-s/faq.md) + +## Labeling / Training + +* [Item Labeling](labeling-training/item-labeling.md) +* [Peak vs Off Peak](labeling-training/peak-vs-off-peak.md) + +## Catalog + +* [Overview](catalog/catalog.md) +* [Business Units](catalog/business-units.md) +* [Catalog Bulk Actions](catalog/catalog-bulk-actions.md) + +## Reports + +* [Built-In](reports/built-in/README.md) + * [Completeness Report](reports/built-in/completeness-report.md) + * [Coverage Report](reports/built-in/coverage-report.md) + * [Dataset Report](reports/built-in/profile.md) + * [Summary Reports](reports/built-in/owl-summary-reports.md) +* [Custom](reports/custom.md) + +## APIs + +* [REST APIs](apis/rest-apis/README.md) + * [Find the endpoint](apis/rest-apis/find-the-endpoint.md) + * [Assignment API](apis/rest-apis/assignmentapi.md) + * [Export and Import API](apis/rest-apis/export-and-import-api.md) + * [Export and Import Example](apis/rest-apis/rules-import-export.md) + * [Time Zone API](apis/rest-apis/owl-time-zone-api.md) +* [Swagger](apis/swagger.md) +* [Notebook/Spark](apis/notebook/README.md) + * [Spark-shell Sample](apis/notebook/spark-shell-sample.md) + * [Dupe](apis/notebook/dupe.md) + * [Load](apis/notebook/load.md) + * [Source](apis/notebook/source.md) + * [Profile](apis/notebook/profile.md) + * [Notebook API](apis/notebook/notebook-api/README.md) + * [OwlOptions (base)](apis/notebook/notebook-api/owloptions-base.md) + * [JWT](apis/notebook/jwt.md) + * [Cookie](apis/notebook/cookie.md) + +## Solutions + +* [Our Approach](projects/why-owl/README.md) + * [Our Story](projects/why-owl/our-story.md) + * [What is OwlDQ](projects/why-owl/what-is-owldq.md) + * [DQ is the difference](projects/why-owl/data-quality.md) +* [Best Practices](projects/best-practices/README.md) + * [Prescriptive Personas](projects/best-practices/prescriptive-personas.md) +* [Data Projects](projects/data-projects/README.md) + * [Builds a Better DQ Dashboard](projects/data-projects/builds-a-better-dq-dashboard.md) + * [Ensures CCPA & GDPR](projects/data-projects/ccpa-and-gdpr.md) + * [Makes your Data Lake better.](projects/data-projects/data-quality-monitoring.md) + * [Speeds Migrations/Enables Replications](projects/data-projects/migrations.md) + * [Assists Data Aggregation](projects/data-projects/assists-data-aggregation.md) + * [Creating a Data Quality Pipeline](projects/data-projects/creating-a-data-quality-pipeline.md) +* [Use Cases](projects/use-cases/README.md) + * [Bank Loans](projects/use-cases/bank-loans.md) + * [Bloomberg Data](projects/use-cases/bloomberg-data.md) + * [Cyber Anomalies in Real-Time](projects/use-cases/cyber-anomalies-in-real-time.md) + * [Financial FxRate Data](projects/use-cases/financial-fxrate-data.md) + * [Healthcare Data Quality](projects/use-cases/healthcare-dq-in-2-minutes.md) + * [Health Insurance Claims Data](projects/use-cases/insurance-data.md) + * [Intraday Positions](projects/use-cases/intraday-positions.md) + * [Security Reference Data](projects/use-cases/security-reference-data.md) + * [Smart Meter Data](projects/use-cases/smart-meter-data.md) + * [Validating Data Movement](projects/use-cases/copying-or-moving-data.md) + +## Benchmarks + +* [Performance Settings](benchmarks/performance-settings.md) +* [Performance Tests](benchmarks/performance-tests.md) +* [Performance Tuning](benchmarks/performance-tuning/README.md) + * [Performance Considerations](benchmarks/performance-tuning/benchmark-faq.md) + +## Architecture + +* [Architecture Diagram](architecture/diagram/README.md) + * [ERD](architecture/diagram/erd.md) +* [System Requirements](architecture/system-requirements/README.md) + * [Hardware Sizing](architecture/system-requirements/hardware-sizing.md) + +## Admin + +* [Overview](admin/admin-console-overview.md) +* [Configuration](admin/configuration/README.md) + * [Multi-Tenancy](admin/configuration/multi-tenancy.md) + * [Time Based Data Retention](admin/configuration/time-based-data-retention.md) + * [Set up SMTP](admin/configuration/smtp-setup.md) + * [Advanced](admin/configuration/custom-configurations.md) +* [Audit](admin/audit/README.md) + * [Dataset Audit Trail](admin/audit/dataset-audit-trail.md) + * [Security Audit Trail](admin/audit/security-audit-trail.md) + * [User Audit Trail](admin/audit/user-audit-trail.md) + +## Security + +* [Overview](security/owl-security.md) +* [Configuration](security/configuration/README.md) + * [Active Directory LDAP](security/configuration/authentication-with-active-directory-ldap/README.md) + * [AD Group to Owl Role Mapping](security/configuration/authentication-with-active-directory-ldap/ad-group-to-owl-role-mapping.md) + * [Connection Security](security/configuration/connection-security.md) + * [Dataset Security](security/configuration/dataset-security/README.md) + * [Dataset Masking](security/configuration/dataset-security/dataset-masking.md) + * [Local User Store Authentication](security/configuration/authentication-with-local-user-store/README.md) + * [Adding Local Users](security/configuration/authentication-with-local-user-store/adding-local-users.md) + * [Role Based Access Control (RBAC)](security/configuration/role-based-access-control-rbac.md) + * [SAML Authentication](security/configuration/authentication-with-saml-idp/README.md) + * [Multi-tenancy support through SAML RelayState](security/configuration/authentication-with-saml-idp/multi-tenancy-support-through-saml-relaystate.md) + * [Securing Passwords](security/configuration/securing-passwords.md) + * [SSL Setup (HTTPS)](security/configuration/ssl-setup-https.md) diff --git a/admin/admin-console-overview.md b/admin/admin-console-overview.md new file mode 100644 index 00000000..52b59cbe --- /dev/null +++ b/admin/admin-console-overview.md @@ -0,0 +1,28 @@ +# Overview + +![](<../.gitbook/assets/image (147).png>) + +The left hand icons “Dashboard”, “Inventory”, “Connections”, and “Configuration” are shortcut links to different operations within Owl. + +* “Dashboard” (See Screenshot below) + * shows an overall number of Owlcheck scanned the MB’s, total jobs, total records scanned. + * Total number of datasets - including passing jobs and jobs that failed (quality fails), # of alerts, # of rules + * A list of messages on specific Owlchecks and what was found. + +![](https://lh3.googleusercontent.com/jHwoAmIsWNmkC3QKQkCaJkqZ5qth2qvylinzDsaWpxBBfqd\_PZBWdgoMDyXnNF1imDUAJP-ub9oux7yxA\_oQ2MSG22qrmR47Ud-Qx5ToY0JgaknBa0eqoG2BJrbutb5GU0BiZO6A) + +* “Inventory” + * Inventory of all Owlcheck ever executed, including (Run Date, Dataset, Command Line, Type, Query, Connection). + * See screenshot below. + +![](https://lh6.googleusercontent.com/vfrlHGmcM\_jTbf1laf44DBZHqfWnIzPnX3Zf2iryxakcqgSkomxpygRaU0spfOjxS6PWHUWNPhtADajZQg2WZAvMQ0PHlgyqCRlhAFiz2ekwkpIITgD6sU6k6p2EdfZqMMuYYK6u) + +* “Connections” - See Owl Build in the “DB Connection” section of this document. +* “Configuration” - Administrator overrides. + * Limit the amount of DataShapes recorded (as an example) on a specific dataset. + * If a duplicate record is found have a 1 point (negative) score per record. Allow the admin to increase or decrease the impact of different DQ issues found. + * Display limits in order to not overwhelm the UI. +* Audit Trail” - + * List all security related changes by user, action, description and timestamp in a searchable sortable table + +![](https://lh3.googleusercontent.com/6uQay257hLCwrZ7VvlmXsBPt-4U1EHfvEbR8cPcLdziC\_035kz5fdCVJ5ZbW5XyS7iX7bga\_L6z6f9sZl7imqKCu8i7VL2lAgKaH9jB\_-3pEjYuJUoo7IJWqyzUf6SRc3OkU9BEW) diff --git a/admin/audit/README.md b/admin/audit/README.md new file mode 100644 index 00000000..5952d526 --- /dev/null +++ b/admin/audit/README.md @@ -0,0 +1,3 @@ +# Audit + +Built-in auditing allows you to track usage and modifications across dataset, security, and user levels. diff --git a/admin/audit/dataset-audit-trail.md b/admin/audit/dataset-audit-trail.md new file mode 100644 index 00000000..aaf482e3 --- /dev/null +++ b/admin/audit/dataset-audit-trail.md @@ -0,0 +1,13 @@ +# Dataset Audit Trail + +Available Data + +* User +* Dataset +* Selected Features +* runID +* Assignments +* Comments +* Timestamp + +![](<../../.gitbook/assets/image (65).png>) diff --git a/admin/audit/security-audit-trail.md b/admin/audit/security-audit-trail.md new file mode 100644 index 00000000..9b825d7f --- /dev/null +++ b/admin/audit/security-audit-trail.md @@ -0,0 +1,11 @@ +# Security Audit Trail + +Available Data + +* User profile updates +* Role updates +* Reference table from metastore +* Dataset deletion requests +* Job schedule attempts + +![](<../../.gitbook/assets/image (64).png>) diff --git a/admin/audit/user-audit-trail.md b/admin/audit/user-audit-trail.md new file mode 100644 index 00000000..00bbd43c --- /dev/null +++ b/admin/audit/user-audit-trail.md @@ -0,0 +1,12 @@ +# User Audit Trail + +Available Data + +* Logins: Successful / Failed +* Privileged User Access +* User Actions / Activities +* Account Actions +* Source IP +* Timestamps + +![](<../../.gitbook/assets/image (63).png>) diff --git a/admin/configuration/README.md b/admin/configuration/README.md new file mode 100644 index 00000000..94c18d7e --- /dev/null +++ b/admin/configuration/README.md @@ -0,0 +1,2 @@ +# Configuration + diff --git a/admin/configuration/custom-configurations.md b/admin/configuration/custom-configurations.md new file mode 100644 index 00000000..545889e6 --- /dev/null +++ b/admin/configuration/custom-configurations.md @@ -0,0 +1,19 @@ +--- +description: /a/appconfig +--- + +# Advanced + +Allow/Disallow Using the catalog feature from Wizard: BULK\_CATALOG\_ON = **TRUE**/FALSE + +Preset Wizard Config for showing views: DB\_VIEWS\_ON = **TRUE**/FALSE + +Preset Wizard Config for showing stats: DB\_STATS\_ON = **TRUE**/FALSE + +Set File Search Path From Wizard: UPLOAD\_PATH = \ + +Set/Unset -p/-srcp masking in UI: MASK\_P\_FLAG = **TRUE**/FALSE + +Allow/Disallow Zeppelin Notebooks: ZEPPELIN\_ENABLED = **TRUE**/FALSE + +Allow/Disallow Using the orient DB: ORIENT\_ENABLED = **TRUE**/FALSE diff --git a/admin/configuration/multi-tenancy.md b/admin/configuration/multi-tenancy.md new file mode 100644 index 00000000..bb48e1a8 --- /dev/null +++ b/admin/configuration/multi-tenancy.md @@ -0,0 +1,91 @@ +--- +description: Divide and conquer your data quality +--- + +# Multi-Tenancy + +MultiTenancy allows a company the ability to instantiate different organizations within one entity. For example say your organization is called Acme and inside of Acme there are two divisions AcmeTraders and AcmeInsurance and each organization is not allowed to see one another's owlcheck results. You would simple segregate them into 2 different Tenants within the overarching Owl web application. + +**Prerequisite(s)** + +* DNS entry for owl web server IP = example below we call it hub (existing for single tenant setup) +* DNS entry for multiTenantSchemaHub = example below we call it owlhub.hub + +**Prerequisites For URL Based tenancy: tenant.host** + +A DNS entry for each tenant + +* DNS entry for every tenant you want to create = example below we call it tenant1.hub + +Each record above points to the same IP address. + +**Setup** + +In order to setup multi-tenancy follow these steps + +* If this is an upgrade please make sure to follow the steps outlined in the "[Upgrading to latest Version](broken-reference)" +* Make sure the web application has started up one time and you successfully logged into it with the default credentials. +* Then stop all the components using ./owlmanage.sh stop +* Modify the owl-env.sh file to include these to new parameters + 1. **export multiTenantSchemaHub=owlhub** (this is a new schema that will get created on owlweb start, note the name of the TenantSchemaHub can be changed to the desired name at setup time) + 2. **export MULTITENANTMODE=TRUE** (this enabled multi-tenancy to be used). + 3. **export URLBASEDMULTITENANTMODE=TRUE/FALSE** + 1. TRUE (default) means you are using the tenant name as a sub-domain (see prerequisites) + 2. FALSE means you will let owl manage tenants via sessions/tokens +* If using agents as part of the operation of owl please be sure to modify the owl.properties file to include the following. + 1. spring.agent.datasource.url=jdbc:postgresql://cdh-edge-dan.us-east4-c.c.owl-hadoop-cdh.internal:5432/postgres**?currentSchema=owlhub** (matching the name of the schema set on step 3-1 above). + 2. jdbc:postgresql://cdh-edge-dan.us-east4-c.c.owl-hadoop-cdh.internal:5432/postgres**?currentSchema=owlhub** (matching the name of the schema set on step 3-1 above). +* Once the settings have been configured for multi-tenancy please start up the owlweb host first using ./owlmanage.sh start=owlweb. Once the web is up and you can hit the page please start up the agents using ./owlmanage.sh start=owlagent. +* In order to use multi-tenancy in URLBASEDMULTITENANTMODE=TRUE you'll have to make sure we have DNS entries to the tenant endpoints, otherwise click the tenant management link from the login page. Example: + 1. If I have a DNS alias named hub. I should be able to point me browser at hub:9002 (or your respective owlweb port) to get to the main Multi-Tenant login page as depicted below + +![](<../../.gitbook/assets/image (82).png>) + +* This is where DNS alias come into place. Assuming we left the owlhub as the multiTenantSchemaHub name we hit the drop down and select owlhub and click the arrow it will place owlhub.hub into the url. This means there also has to be a DNS Alias name for your selected multiTenantSchemaHub name. NOTE: Username and password for tenant management is mtadmin / mtadmin123 + +![](<../../.gitbook/assets/image (83).png>) + +* Now that you logged into the Tenant Management screen using the hub DNS alias we can create our first tenant. In this example below I'm going to create a tenant named tenant1. First click the "+ Add Tenant" button in the top right part of the screen. + +![](<../../.gitbook/assets/image (84).png>) + +Click Save. Your tenant shows up in the list and now you can click the login button as shown below. + +![](<../../.gitbook/assets/image (85).png>) + +Clicking the Login button will redirect your browser to the tenant1.hub:9002 url (DNS entry needs to be in place for tenant1 as shown below). + +![](<../../.gitbook/assets/image (86).png>) + +Enter the admin username and password that you created for the tenant1 (refer to figure 3 about) and login to the tenant as the admin. + +While logged in as a tenant admin the last step is to go to the Admin Console and click on "Sync Schema" this will generate the tables under the tenant called tenant1. + +![](<../../.gitbook/assets/image (87).png>) + +At this point you are ready to start administrating your tenant1 as you did with the owl web application in the past. + +### Supplemental: Adding and Editing a Tenant + +In many cases it may make sense to have isolated environments to check data quality. This need could be driven by a number of factors including data access rights, organization and business models, reporting needs, and/or other security requirements. + +Regardless of the need, Owl will support dynamically creating tenants via our Owl Hub Management portal as part of the Owl Web Application. That's it, there is nothing else to install, simply enable Multi-Tenant mode in the application configuration properties and you are on your way. + +Once enabled you will have a tenant selection screen prior to login where you can chose any of your configured tenants or access the Owl Hub (with the TENANT\_ADMIN role) + +![](<../../.gitbook/assets/Screen Shot 2019-09-03 at 11.34.13 AM.png>) + +After selecting the owlHub tenant, you will have the ability to manage each tenant, as well as create new tenants from the management console. + +![](<../../.gitbook/assets/Screen Shot 2019-09-03 at 10.51.28 AM.png>) + +All enabled tenants will be listed in the multi-tenant drop down menu. Access to tenants are handled by the administrator(s) within each tenant individually. + +Access to agents are also handled by the administrator(s) within each tenant individually. + +![](<../../.gitbook/assets/Screen Shot 2019-09-04 at 12.35.34 PM.png>) + +Each agent is visible and editable as an Admin from the UI. + +![](<../../.gitbook/assets/owl-agent (1) (1).png>) + diff --git a/admin/configuration/smtp-setup.md b/admin/configuration/smtp-setup.md new file mode 100644 index 00000000..15bf2ff8 --- /dev/null +++ b/admin/configuration/smtp-setup.md @@ -0,0 +1,57 @@ +# Set up SMTP + +Simple Mail Transport Protocol or SMTP, is an internet standard for email transmission. Collibra DQ allows you to configure a single SMTP server to send alerts to the attention of the dataset owner in case a specific condition is met, such as: + +* Owl Data Quality Score is below a specific threshold. +* Row Count is below a specific threshold. +* If a rule is triggered. + +### Steps + +1. In the **Admin Console**, click **Alerts**. +2. In the **Configuration** section, enter the required information: + +| Option | Description | +| ------------------ | -------------------------------------------------------------------------- | +| SMTP Host | The name or the IP address of the SMTP server. | +| SMTP Port | The port used by the SMTP server. | +| SMTP Username | The username or the account that is configured on the SMTP server for use. | +| SMTP Password | The password of the SMTP server username or account. | +| (Default) To Email | The sender email address. | +| Reply Email | The reply-to email address. | + +Save your changes. + +![SMTP setup in Collibra DQ Admin Console](../../.gitbook/assets/smtp\_setup.gif) + +See screenshot below for example of configuration.![](http://18.204.201.140:8080/xwiki/bin/download/Documentation/Admin%20Guide/Setting%20Up%20SMTP/WebHome/1555069327842-964.png)Configuring SMTP + +When completed click the “Add” button. + +Once the information has been populated and added, the grey box above the form will get populated with the content supplied. If the data ever has to be changed clicking the ![1555069526616-123.png](http://18.204.201.140:8080/xwiki/bin/download/Documentation/Admin%20Guide/Setting%20Up%20SMTP/WebHome/1555069526616-123.png?width=35\&height=31) icon will repopulate the form in order to be modified and re added. + +Now that configuration of the SMTP Email Server has been completed let’s create an alert and see that the alert triggers an email. In this example, we will use the dist\_example dataset that we ran earlier from the demo.sh script. + +![](<../../.gitbook/assets/image (90).png>) + +In the above screen shot we did the following: + +1. Searched for the “dist\_example” dataset +2. Provided an alert named “score\_lt\_90” +3. Provided a condition that we know will be met “score < 90” +4. Provided who the recipient of this alert should be mailed to in this sample “user2@owl.net” +5. Custom Message = “score is below 90 for dist\_exampe” + +Clicking the “Save” button will move the contents of the form above to the List of Alerts for this particular dataset. + +From the terminal on this install if you run the below command (which is just an extract out of the demo.sh file). We should see the alert get triggered + +./owlcheck -ds dist\_example -rd 2018-10-07 -d , -f /opt/owl/bin/demos/distribution\_change.csv -fq "select \* from dataset where d\_date ='2018-10-07'" + +At the end of this command we should see the “Alert was triggered” as shown in the screenshot below. + +![](<../../.gitbook/assets/image (89).png>) + +And the recipient [user2@owl.net](mailto:user2@owl.net) received the email. + +![](<../../.gitbook/assets/image (88).png>) diff --git a/admin/configuration/time-based-data-retention.md b/admin/configuration/time-based-data-retention.md new file mode 100644 index 00000000..2e8efc8e --- /dev/null +++ b/admin/configuration/time-based-data-retention.md @@ -0,0 +1,82 @@ +--- +description: Setup and Configuration +--- + +# Time Based Data Retention + +## **Setting up Retention Based Data Purge** + +Retention based purge of data can be turned on to allow data to automatically be cleaned based on an organization's data retention policy. + +### **Benefit** + +Once enabled, what type of data is removed? + +* Dataset statistics +* Data\_Preview +* Dataset\_Field +* Rule\_Breaks +* Job Ledger + +### **Setup** + +In order to set up retention based data purge, three (3) environment variables need to be set up in the owl-env.sh configuration script. Note: a restart of the webapp is required for this configuration to take place. + +* **cleaner\_retention\_enabled** + * TRUE or FALSE on whether this feature is enabled +* **cleaner\_retention\_days** + * Number of days to retain data +* **cleaner\_retention\_field** + * Controls which field to use to select eligible dataset runs + * Potential values + * updt\_ts: consider the last time a dataset run was updated + * run\_id: consider the run id field of the dataset + +### Configuration + +**Example configuration in owl-env.sh** + +Organization wants to purge data where the updt\_ts is more than 1 year old + +**In owl-env.sh, add the following lines** + +``` +export cleaner_retention_enabled=TRUE +export cleaner_retention_days=365 +export cleaner_retention_field="updt_ts" +``` + +### Config Map + +``` +autoClean: "false" +cleaner_retention_days: "180" +cleaner_retention_field: updt_ts +cleaner_retention_enabled: "true" +``` + +### Defaults for Auto Clean Process + +{% hint style="info" %} +This is a separate rolling purge that is **distinct from the time-based retention.** This is **on by default** and uses the pre-defined limits below. You will see audit records for this clean-up process in the audit history of the admin console. +{% endhint %} + +Separate from the time-based retention there is also a default auto clean mechanism that actively purges your old records. This is enabled by default and can be modified by use of the autoClean (AUTOCLEAN) boolean parameter. + +``` +AUTOCLEAN=false or autoClean="false" + +### Depending whether this is part of owl-env.sh +### or the configMap of the web pod +``` + +These are the defaults. The row count threshold is the global limit when this is triggered. This is based on the records in the data\_preview table. The runs threshold and the dataset per row threshold are dataset-level limits that require a dataset to have at least 4 scans and at least 1000 rows. + +This is an example using the owl-env.sh file to control these settings. + +``` +export AUTOCLEAN=true +export DATASETS_PER_ROW=1000 +export RUNS_THRESHOLD=4 +export ROW_COUNT_THRESHOLD=200000 +``` diff --git a/apis/notebook/README.md b/apis/notebook/README.md new file mode 100644 index 00000000..a3d37a7f --- /dev/null +++ b/apis/notebook/README.md @@ -0,0 +1,2 @@ +# Notebook/Spark + diff --git a/apis/notebook/cookie.md b/apis/notebook/cookie.md new file mode 100644 index 00000000..037a364a --- /dev/null +++ b/apis/notebook/cookie.md @@ -0,0 +1,22 @@ +--- +description: Use cookies file to Run Owl CURL Commands +--- + +# Cookie + + + +```bash + +curl -i -X POST -d username= -d password= http://localhost:9000/login -c cookies.txt + +curl -i --header "Accept:application/json" -X GET -b cookies.txt "http://localhost:9000/v2/getsecuritymap" +``` + +Multi-Tenant without subdomain in URL (tenant parameter required): + +```bash +curl -i -X POST -d username= -d password= -d tenant= -d tenant=public http://localhost:9000/login -c cookies.txt + +curl -i --header "Accept:application/json" -X GET -b cookies.txt "http://localhost:9000/v2/getsecuritymap" +``` diff --git a/apis/notebook/dupe.md b/apis/notebook/dupe.md new file mode 100644 index 00000000..e882ae90 --- /dev/null +++ b/apis/notebook/dupe.md @@ -0,0 +1,89 @@ +# Dupe + + + +```java +package com.owl.common.options; + +/** + * Options for Dupe Activity + */ +public class DupeOpt { + + /** + * Whether to run Dupe Activity + */ + public Boolean on = false; // --dupe + + /** + * @deprecated Unused for Activity2 + */ + public Boolean only = false; // --dupeonly + + /** + * Column names to include Dupe Activity + */ + public String[] include; // -dupeinc + + /** + * Column names to exclude Dupe Activity + */ + public String[] exclude; // dupeexc + + /** + * Indicator for complexity. See Activity2.Dupe.Scala.execute() + * depth == 0 : exact match (sets props.dupeExactMatch = TRUE downstream) + */ + public Integer depth = 2; // -depth + + /** + * The minimum dupe scores between two duplicates. (currently calculated as "edit distance", out of upperBound) + * Two values with dupe score less than this is lowerBound are not duplicates (i.e. "truly" different values) + */ + public Integer lowerBound = 80; // -dupelb, -dupecutoff + + /** + * The maximum possible dupe score for duplicate records (for a given dupe detection method). + * Currently assumed to be 100. + */ + public Integer upperBound = 100; // -dupeub, -dupepermatchupperlimit + + /** + * Approximate dupe score used to create block index (when DF is large) + */ + public Integer approximate = 1; // -dupeapprox + + /** + * Number of observations per unique duplicate + */ + public Integer limitPerDupe = 15; + + /** + * Whether to process column headers when data load uses manual column names (LoadOpts.fileHeader) + * TODO this belongs in LoadOpts, not DupeOpts + */ + public Boolean checkHeader = true; + + /** + * TODO remove + * + * @deprecated not used; + */ + public String filter; + + /** + * If true, dupe activity is case insensitive. If false, dupe activity is case sensitive. + * Convenience feature for upper and lower set to 100 + */ + public Boolean ignoreCase = false; //-dupenocase + + /** + * Number of points each duplicate contributes to the total schema score (in Hoot) + */ + public Double score = 1.0; //-dupescore points per duplicate found default 1 + + /** + * Number of unique duplicates to compute during dupe activity + */ + public Integer limit = 300; //-dupelimit default 300 +``` diff --git a/apis/notebook/jwt.md b/apis/notebook/jwt.md new file mode 100644 index 00000000..c8cbabc8 --- /dev/null +++ b/apis/notebook/jwt.md @@ -0,0 +1,30 @@ +--- +description: Use JSON Web Tokens to Run Owl CURL Commands +--- + +# JWT + +```bash +TOKEN=$(curl -s -X POST http://localhost:9000/auth/signin -H "Content-Type:application/json" -d "{\"username\":\"\", \"password\":\"\"}" | jq -r '.token') + +curl -i -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" http://localhost:9000/v2/getsecuritymap + +``` + +Multi-Tenant without subdomain in URL (tenant parameter \[iss] required): + +```bash +TOKEN=$(curl -s -X POST http://localhost:9000/auth/signin -H "Content-Type:application/json" -d "{\"username\":\"\", \"password\":\"\", \"iss\":\"\"}"| jq -r '.token') + +curl -i -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" http://localhost:9000/v2/getsecuritymap + +``` + + + +Without Headers and jq display: + +```bash +curl -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" http://localhost:9000/v2/getsecuritymap | jq '.' | cat +``` + diff --git a/apis/notebook/load.md b/apis/notebook/load.md new file mode 100644 index 00000000..f50d7964 --- /dev/null +++ b/apis/notebook/load.md @@ -0,0 +1,386 @@ +# Load + +```java +package com.owl.common.options; + +import org.apache.commons.lang3.StringUtils; + +import java.util.Properties; + +/** + * Owl Options related to data loading + */ +public class LoadOpt { + // Options order: "unsorted", + // "dataset scope columns", "dataset scope rows", "look back", + // "common options for both data sources", "file as data source", "db as data source" + + public static final String SINGLE_QUOTE = "'"; + public static final String DOUBLE_QUOTE = "\""; + public static final String BACK_TICK = "`"; + + /** + * If true, don't save any metadata + * TODO confirm if this is correct + */ + public Boolean readonly = false; + + /** + * The Password manager. + */ + public String passwordManager = null; + + /** + * Catalog alias (Catalog name) + */ + public String alias = StringUtils.EMPTY; + + // --- Dataset Scope Column specifications ------- // + // Properties that select columns for Dataset activities or modifies (data type or new columns) + // prior and/or during loading into Spark DF + /** + * Dataset scope query. (IMPORTANT) + * The query should contain all the columns necessary to run the activities. + * TODO: figure out if this gets used when using files + */ + public String query = StringUtils.EMPTY; + + /** + * Concatenated column names (sep = ",") for columns that are keys + * TODO: confirm + */ + public String key = StringUtils.EMPTY; + + /** + * SELECT expression to transform expressions with assignment by "=" and delimited by "|". + * e.g. colname=cast(colname as string)|colname2=colname2(cast as date) + */ + public String expression = StringUtils.EMPTY; + + /** + * Add "OWL_RUN_ID" UNIX timestamp (s) column to Spark DF usng the OwlOptions.runId. + * Does not obey timeStampDivisor (timestamp in seconds because Spark) + */ + public Boolean addDateColumn = false; + + /** + * Fill null values in Spark DF with 0 (numeric columns only) + */ + public Boolean zeroFillNull = false; + + /** + * A string that indicates a null value; any value matching this string will be set as nulls in the Spark DF + * Default: "" -> NULL + * Example: 'null' -> NULL + * -- + * Note: to emptyStirngFillNull (replace String column null -> "", use expression + */ + public String replaceNulls = StringUtils.EMPTY; + + /** + * All data types forced to strings for type safe processing. + * Not implemented in activity (yet) + */ + public Boolean stringMode = false; + + // --- Dataset Scope Row specifications ------- // + // Properties that filter rows for Dataset activities + // prior and/or during loading into Spark DF + /** + * Convert row into string and only use rows containing this value. + * Strict matching only. + */ + public String filter = StringUtils.EMPTY; + + /** + * Convert row into string and only use rows containing this value. + * Strict matching only. + */ + public String filterNot = StringUtils.EMPTY; + + // --- Look back ------- // + // For Look back feature + /** + * Build up history of OwlChecks. Does not include current OwlCheck. + * TODO: Document the relationship with unionLookBack + */ + public Integer backRun = null; + + /** + * Whether to load data for looking back in history. + * How much historical data to load is based on OutlierOpt.lookback and PatternOpt.lookback. + */ + public Boolean unionLookBack = false; + + // --- Shared Data Loading Options ------- // + // Properties that affect data loading & pre-processing for both files and db as source + /** + * Whether to use cached data for activities + */ + public Boolean cache = true; + + /** + * The year, month, and day format of date columns in the dataset for loading the data only. + * Default = "yyyy-MM-dd" + */ + public String dateFormat = "yyyy-MM-dd"; + + /** + * The hour, minute, second, and milisecond format of date columns in the dataset for loading the data only/ + * Default = "HH:mm:ss.SSS" + * Not used. Questionably why separate timeFormat variable exists when dateFromat can represent hms as well. + */ + public String timeFormat = "HH:mm:ss.SSS"; + + /** + * Whether to convert date columns (specified by activity opts) in dataset + * into timestamp in ms (to make it seconds, set Props.timeStampDivisor = "s") + * TODO: Needs LoadOpt.timeStampDivisor and fix Utils.scala date2Timestamp + */ + public Boolean timestamp = false; + + /* TODO add timeStampDivisor here and map between owl props? + public String timeStampDivisor = "ms" + */ + + // --- Using file as data source ------- // + // Properties that control where & how static file is read + /** + * Full path to the file. + * If hdfs, then "hdfs://...". + * If s3, then "s3://...", "s3a://...", or "s3n://...". + * If parquet, then "...parquet" or "...PARQUET" + */ + public String filePath = StringUtils.EMPTY; + + /** + * SQL query used on file. + * owl_id is added if not included in select clause. + * If empty, then defaults to full file query. + * (Does not update LoadOpts.fullFile to true). + */ + public String fileQuery = StringUtils.EMPTY; + + /** + * Whether to use full file (i.e. use all columns) on data load + */ + public Boolean fullFile = false; + + /** + * File column names, comma separated + */ + public String fileHeader = null; + + /* TODO checkHeader needs to be moved here from DupeOpt + public Boolean checkHeader = true;*/ + + /** + * Whether to have Spark infer the schema of data source + * If props.profile2 == true, this is overwritten to false! + * If xml file, this is ignored and schema is always inferred by Spark on xml data load. + * If avro file, this value is respected (but may get overwritten by props.profile2) + * (see activity2.Load.file) + */ + public Boolean inferSchema = true; + + /** + * Sample without replacement from file. Valid value is a fraction [0, 1.0]. + * Only affects when filetype is xml or unspecified (and therefore assumed to be delimited table) + */ + public Double sample = 1.0; + + /** + * Filetype (avro, json, orc, parquet, xml). Unspecified file + */ + public FileType fileType = null; + + /** + * Delimiter for file. If number of characters after replacing "\" with "" is 2 or more character + * (e.g. compound delimiters like \t\t), then defaults to "\t" and attempts to read file as tsv + * See Activity2.load.file for details + */ + public String delimiter = ","; + + /** + * File character encoding + */ + public String fileCharSet = "UTF-8"; + + /** + * The Avro schema for relevant avro file. Ignored if empty string + */ + public String avroSchema = StringUtils.EMPTY; + + /** + * The Xml row tag for xml file. Ignored if empty string. + */ + public String xmlRowTag = StringUtils.EMPTY; + + /** + * Whether to flatten arrays in nested schema + * TODO explain better. Does this only affect JSON file? + */ + public Boolean flatten = false; + + /** + * Whether data contains maps in json that requires extra handling" + * TODO explain better. Does this only affect JSON file? + */ + public Boolean handleMaps = false; + + /** + * Whether to handle mixed json. + * TODO explain better. Does this only affect JSON file? + */ + public Boolean handleMixedJson = false; + + /** + * Spark.read option multiline, for JSON file only + */ + public Boolean multiLine = false; + + // --- Using database as data source ------ // + /** + * Path to DB Driver. (e.g. /opt/owl/driver/postgres) + */ + public String lib = StringUtils.EMPTY; + + /** + * DB Driver name (Java namespace, e.g. org.postgresql.Driver). + * Leave as null (default) and LoadOpts.connectionURL will resolve the driver name. + */ + public String driverName = null; + + /** + * Connections name in metastore DB (public.connections.aliasname). + * Does not refer to the "name" of the database. Refers to "aliasname" that the user set when + * uploading connection config to Owl. + */ + public String connectionName = StringUtils.EMPTY; + + /** + * The Connection url, prefixed by jdbc. + * e.g. "jdbc:postgresql://localhost:5432" + */ + public String connectionUrl = StringUtils.EMPTY; + + /** + * DB username + */ + public String userName = StringUtils.EMPTY; + + /** + * DB password + */ + public String password = StringUtils.EMPTY; + + /** + * JDBC Connection properties (e.g. fetchsize) + */ + public Properties connectionProperties = null; + + /** + * Whether data source is Hive Native (not using JDBC) + * TODO: Why is the default null as opposed to false? + */ + public Boolean hiveNative = null; + + /** + * Whether data source is Hive Hadoop Web Cluster (not using JDBC) + */ + public Boolean hiveNativeHWC = false; + + // --- Parallel JDBC ------- // + /** + * When running parallel JDBC, use LoadOpts.query and OwlOptions.dataset as base table + */ + public Boolean useSql = true; + + /** + * When running parallel JDBC, specify column name + * ?? Activity2.Load and web has hard-coded magic string "OWLAUTOJDBC" + */ + public String columnName = null; + + /** + * When running parallel JDBC, the upper bound for partition column. + * (e.g. "1000000") + */ + public String lowerBound = null; + + /** + * When running parallel JDBC, the upper bound for partition column. + * (e.g. "5000000") + */ + public String upperBound = null; + + /** + * When running parallel JDBC, the number of partitions used. + * If 0, then numPartitions used is based on the number of available Spark Executor (1/2 ~ 2/3) + * If > 20, then overwritten to 20 (no more than 20 concurrent connections to a database on a single dataset) + */ + public Integer numPartitions = 0; + + // --- SQL Query properties ---------- // + // TODO: does this effect DB as source or file as source as well? + /** + * Whether the escape character would be back tick (`). + * Ignored if escapeCharacter is non-empty (if using OwlCheck from Options). + * Marked as true if props.escapeCharacter is a tick + * (to preserve bijection between props and opts, and vice versa). + */ + public Boolean escapeWithBackTick = false; + /** + * Whether the escape character would be single quote ('). + * Ignored if escapeCharacter is non-empty (if using OwlCheck from Options). + * Marked as true if props.escapeCharacter is a tick + * (to preserve bijection between props and opts, and vice versa). + */ + public Boolean escapeWithSingleQuote = false; + /** + * Whether the escape character would be double quote ("). + * Ignored if escapeCharacter is non-empty(if using OwlCheck from Options). + * Marked as true if props.escapeCharacter is a tick + * (to preserve bijection between props and opts, and vice versa). + */ + public Boolean escapeWithDoubleQuote = false; + + /** + * Specify custom escape character. This takes precedence over all other escapeWithXYZ options. + * i.e. if non-empty, then other escapeWithXYZ options are ignored. + * If empty (default), no escaping attempt is made (and SQL query may fail if it contains reserved word) + * + * @deprecated Access level of this field will be changed to private. Please use {@link #setEscapeCharacter(String)} instead. + */ + @Deprecated + public String escapeCharacter = StringUtils.EMPTY; + + + /** + * The enum File type. + */ + public enum FileType { + /** + * Avro file type. + */ + avro, + /** + * Json file type. + */ + json, + /** + * Orc file type. + */ + orc, + /** + * Parquet file type. + */ + parquet, + /** + * Xml file type. + */ + xml + } +} + +``` diff --git a/apis/notebook/notebook-api/README.md b/apis/notebook/notebook-api/README.md new file mode 100644 index 00000000..f89d241a --- /dev/null +++ b/apis/notebook/notebook-api/README.md @@ -0,0 +1,6 @@ +--- +description: Using OwlDQ using Notebook API and OwlOptions +--- + +# Notebook API + diff --git a/apis/notebook/notebook-api/owloptions-base.md b/apis/notebook/notebook-api/owloptions-base.md new file mode 100644 index 00000000..ee244bd0 --- /dev/null +++ b/apis/notebook/notebook-api/owloptions-base.md @@ -0,0 +1,48 @@ +--- +description: API Documentation for Java Class OwlOptions. +--- + +# OwlOptions (base) + +The Class properties are laid out in the following format headers to allow Gitbook indexing. + +### ClassName field\_name + +**field type** | _`default`_ \ +__Description + +## OwlOptions dataset + +**String** | _`StringUtils.Empty` _ \ +__Unique string ID for the OwlCheck Dataset. Cannot contain `"."`, `"-"`, `"#"`, `"@"` \ + + + + + + + + + + +## Example Code + +### Initializing + +{% tabs %} +{% tab title="Scala" %} +```scala +import com.owl.common.options.OwlOptions + +val opts = new OwlOptions() +``` +{% endtab %} + +{% tab title="Java" %} +```java +import com.owl.common.options.OwlOptions + +OwlOptions opts = new OwlOptions(); +``` +{% endtab %} +{% endtabs %} diff --git a/apis/notebook/profile.md b/apis/notebook/profile.md new file mode 100644 index 00000000..e5bc7e26 --- /dev/null +++ b/apis/notebook/profile.md @@ -0,0 +1,28 @@ +# Profile + + + +```java +public class ProfileOpt { + + public Boolean on = true; //Whether to run profile + public Boolean only = false; //Whether to run only profile + public String[] include; //Which columns to include + public String[] exclude; //Which columns to exclude + public Boolean shape = true; //Disable shape detection + public Boolean correlation = null;//On/Off Pearsons Correlation, null=auto + public Boolean histogram = null; //On/Off Histograming, null=auto + public Boolean semantic = null; //On/Off Semantic discovery, null=auto + + public Integer limit = 300; + public Integer histogramLimit = 0; + public Double score = 1.0; //downscore points per Shape issue + public Integer shapeTotalScore = 0; + public Double shapeSensitivity = 0.00; + public Integer shapeMaxPerCol = 0; + public Integer shapeMaxColSize = 0; + public String behavioralDimension = StringUtils.EMPTY; + public String behavioralDimensionGroup = StringUtils.EMPTY; + public String behavioralValueColumn = StringUtils.EMPTY; + public Boolean behaviorScoreOff = false; // disable behavior scoring +``` diff --git a/apis/notebook/source.md b/apis/notebook/source.md new file mode 100644 index 00000000..e3cea019 --- /dev/null +++ b/apis/notebook/source.md @@ -0,0 +1,35 @@ +# Source + + + +```java +public class SourceOpt { + + public Boolean on = false; //-vs + public Boolean only = false; //-sourceonly + public Boolean validateValues = false; //-validatevalues + public Boolean matches = false; //-matches + + public String[] include; //-valinc + public String[] exclude; //-valexc + public String[] key; //valkey + public Map map; // + public Double score = 1.0; // points per validate source found, default 1-5 + public Integer limit = 30; //-valsrclimit + public String delimiter = ","; //-srcdel + public String fileCharSet = "UTF-8"; //-srcencoding + public String filePath = StringUtils.EMPTY; //--srcfile + + public String header = null; //-srcheader + public String dataset = StringUtils.EMPTY; + public String driverName = StringUtils.EMPTY; + public String user = StringUtils.EMPTY; + public String password = StringUtils.EMPTY; + public String passwordManager = StringUtils.EMPTY; + public String connectionName = StringUtils.EMPTY; + public String connectionUrl = StringUtils.EMPTY; + public String query = StringUtils.EMPTY; + public String fileQuery = StringUtils.EMPTY; + public String lib = StringUtils.EMPTY; + public Properties connectionProperties; +``` diff --git a/apis/notebook/spark-shell-sample.md b/apis/notebook/spark-shell-sample.md new file mode 100644 index 00000000..a8a96d92 --- /dev/null +++ b/apis/notebook/spark-shell-sample.md @@ -0,0 +1,59 @@ +# Spark-shell Sample + +./bin/spark-shell --jars /opt/owl/bin/owl-core-trunk-jar-with-dependencies.jar,/opt/owl/drivers/postgres/postgresql-42.2.5.jar --deploy-mode client --master local\[\*] + +Import lib’s, if you get a dependency error, please import a second time. + +import com.owl.core.util.{OwlUtils, Util} + +import com.owl.common.domain2.OwlCheckQ + +import com.owl.common.options.\_ + +Set up connection parameters to the database we want to scan if you don’t already have a dataframe + +val _url_ = "jdbc:postgresql://xxx.xxx.xxx.xxx:xxxx/db?currentSchema=schema"\ +val _connProps_ = _Map_(\ + "driver" -> "org.postgresql.Driver",\ + "user" -> "user",\ + "password" -> "pwd",\ + "url" -> _url_,\ + "dbtable" -> "db.table"\ +) + +Create a new OwlOptions object so we can assign properties + +val opt = new OwlOptions() + +Set up variables for ease of re-use + +val dataset = "nyse\_notebook\_test\_final" + +val runId = "2017-12-18" + +var date = runId + +var query = s"""select \* from \ where \ = '**$**date' """ + +val pgDatabase = "dev"\ +val pgSchema = "public" + +Set OwlOptions values to the metastore + +opt._dataset_ = dataset\ +opt._runId_ = runId\ +opt._host_ = "xxx.xxx.xxx.xxx"\ +opt._pgUser_ = "xxxxx"\ +opt._pgPassword_ = "xxxxx"\ +opt._port_ = s"5432/**$**pgDatabase?currentSchema=**$**pgSchema" + +Create a connection, build the dataframe, register and run + +With inline processing you will already have a dataframe so you can skip down to setting the OwlContext + +val conn = _connProps_ + ("dbtable" -> s"(**$**query) **$**dataset")\ +val df = _spark_.read.format("jdbc").options(conn).load + +val owl = OwlUtils._OwlContext_(df, opt)\ +owl.register(opt)\ +owl.owlCheck diff --git a/apis/rest-apis/README.md b/apis/rest-apis/README.md new file mode 100644 index 00000000..7df0c024 --- /dev/null +++ b/apis/rest-apis/README.md @@ -0,0 +1,636 @@ +# REST APIs + +![](../../.gitbook/assets/APIs.gif) + +All REST APIs are available inside the application under admin section. The APIs can be used against the application in live working mode, which is preferred over documentation of APIs because it means the API works and was tested at compile time versus documentation time. + +### Product API + +The product API is for end-users who want to interact with the official and supported API. You can also generate a client side SDK from the API with 4 steps below. + +![](<../../.gitbook/assets/Screen Shot 2021-08-02 at 5.25.08 PM.png>) + +```bash +#psuedo code example REST API + +dataset = 'public.nyse' +runId = '2021-03-05' + +#SAVE datasetDef +dataset = POST /v3/datasetDefs/ {json_datasetDef} + +#UPDATE datasetDef +dataset = PUT /v3/datasetDefs/ {json_datasetDef} + +#RUN JOB +jobId = POST /v3/jobs/run/{dataset},{runDate} + +#CHECK STATUS +status = /v3/jobs{jobId}/status + +#GET DQ FINDINGS +findings = /v3/jobs/{jobId}/findings +``` + +### Generate Client SDK + +1. Go to [https://editor.swagger.io/](https://editor.swagger.io) +2. Click File Import URL +3. Paste a URL that looks like this [https://\/v2/api-docs?group=Product%20API](https://146.148.84.143/v2/api-docs?group=Product%20API) +4. Click generate client (python, java, scala, C#) + +![](<../../.gitbook/assets/Screen Shot 2021-08-03 at 9.05.13 AM.png>) + +```python +#Python SDK Example + +#GET CMDLINE +cmdLine = get_job_cmdline(dataset) + +#SUBMIT JOB +job_id = run(dataset, run_date) + +#CHECK STATUS +status = get_job_status(job_id) + +#GET DQ ISSUES +status = get_job_findings(dataset, run_date) +``` + +### Dataset Definition + +The JSON for the full dataset definition. It can be more terse to send in the cmdline string of just the variables you use for your DQ Job. + +```bash +-df "yyyy/MM/dd" -owluser -numexecutors 1 -executormemory 1g \ +-f s3a://s3-datasets/dataset.csv -h :5432/dev?currentSchema=public \ +-fq "select * from dataset" -drivermemory 1g -master k8s:// -ds dataset_csv_1 \ +-deploymode cluster -bhlb 10 -rd "2021-04-01" -fullfile -loglevel INFO -cxn s3test5 \ +-sparkprinc user2@CW.COM -sparkkeytab /tmp/user2.keytab +``` + +```javascript +{ + "dataset": "", + "runId": "", + "runIdEnd": "", + "runState": "DRAFT", + "passFail": 1, + "passFailLimit": 75, + "jobId": 0, + "coreMaxActiveConnections": null, + "linkId": null, + "licenseKey": "", + "logFile": "", + "logLevel": "", + "hootOnly": false, + "prettyPrint": true, + "useTemplate": false, + "parallel": false, + "plan": false, + "dataPreviewOff": false, + "datasetSafeOff": false, + "obslimit": 300, + "pgUser": "", + "pgPassword": "", + "host": null, + "port": null, + "user": "anonymous : use -owluser", + "alertEmail": null, + "scheduleTime": null, + "schemaScore": 1, + "optionAppend": "", + "keyDelimiter": "~~", + "agentId": null, + "load": { + "readonly": false, + "passwordManager": null, + "alias": "", + "query": "", + "key": "", + "expression": "", + "addDateColumn": false, + "zeroFillNull": false, + "replaceNulls": "", + "stringMode": false, + "operator": null, + "dateColumn": null, + "transform": null, + "filter": "", + "filterNot": "", + "sample": 1, + "backRun": 0, + "backRunBin": "DAY", + "unionLookBack": false, + "cache": true, + "dateFormat": "yyyy-MM-dd", + "timeFormat": "HH:mm:ss.SSS", + "timestamp": false, + "filePath": "", + "fileQuery": "", + "fullFile": false, + "fileHeader": null, + "inferSchema": true, + "fileType": null, + "delimiter": ",", + "fileCharSet": "UTF-8", + "skipLines": 0, + "avroSchema": "", + "xmlRowTag": "", + "flatten": false, + "handleMaps": false, + "handleMixedJson": false, + "multiLine": false, + "lib": "", + "driverName": null, + "connectionName": "", + "connectionUrl": "", + "userName": "", + "password": "", + "connectionProperties": {}, + "hiveNative": null, + "hiveNativeHWC": false, + "useSql": true, + "columnName": null, + "lowerBound": null, + "upperBound": null, + "numPartitions": 0, + "escapeWithBackTick": false, + "escapeWithSingleQuote": false, + "escapeWithDoubleQuote": false, + "escapeCharacter": "", + "hasHeader": true + }, + "outliers": [ + { + "id": null, + "on": false, + "only": false, + "lookback": 5, + "key": null, + "include": null, + "exclude": null, + "dateColumn": null, + "timeColumn": null, + "timeBin": "DAY", + "timeBinQuery": "", + "categorical": true, + "by": null, + "limit": 300, + "minHistory": 3, + "historyLimit": 5, + "score": 1, + "aggFunc": "", + "aggQuery": "", + "query": "", + "q1": 0.15, + "q3": 0.85, + "categoricalColumnConcatenation": false, + "limitCategorical": null, + "measurementUnit": "", + "multiplierUpper": 1.35, + "multiplierLower": 1.35, + "record": true, + "filter": null, + "combine": true, + "categoricalConfidenceType": "", + "categoricalTopN": 3, + "categoricalBottomN": 2, + "categoricalMaxConfidence": 0.02, + "categoricalMaxFrequencyPercentile": 0.25, + "categoricalMinFrequency": 1, + "categoricalMinVariance": 0, + "categoricalMaxCategoryN": 1, + "categoricalParallel": true, + "categoricalAlgorithm": "", + "categoricalAlgorithmParameters": {} + } + ], + "outlier": { + "id": null, + "on": false, + "only": false, + "lookback": 5, + "key": null, + "include": null, + "exclude": null, + "dateColumn": null, + "timeColumn": null, + "timeBin": "DAY", + "timeBinQuery": "", + "categorical": true, + "by": null, + "limit": 300, + "minHistory": 3, + "historyLimit": 5, + "score": 1, + "aggFunc": "", + "aggQuery": "", + "query": "", + "q1": 0.15, + "q3": 0.85, + "categoricalColumnConcatenation": false, + "limitCategorical": null, + "measurementUnit": "", + "multiplierUpper": 1.35, + "multiplierLower": 1.35, + "record": true, + "filter": null, + "combine": true, + "categoricalConfidenceType": "", + "categoricalTopN": 3, + "categoricalBottomN": 2, + "categoricalMaxConfidence": 0.02, + "categoricalMaxFrequencyPercentile": 0.25, + "categoricalMinFrequency": 1, + "categoricalMinVariance": 0, + "categoricalMaxCategoryN": 1, + "categoricalParallel": true, + "categoricalAlgorithm": "", + "categoricalAlgorithmParameters": {} + }, + "pattern": { + "id": null, + "only": false, + "lookback": 5, + "key": null, + "dateColumn": null, + "include": null, + "exclude": null, + "score": 1, + "minSupport": 0.000033, + "confidence": 0.6, + "limit": 30, + "query": "", + "filter": null, + "timeBin": "DAY", + "on": false, + "match": true, + "lowFreq": false, + "bucketLimit": 450000, + "deDupe": true + }, + "patterns": [ + { + "id": null, + "only": false, + "lookback": 5, + "key": null, + "dateColumn": null, + "include": null, + "exclude": null, + "score": 1, + "minSupport": 0.000033, + "confidence": 0.6, + "limit": 30, + "query": "", + "filter": null, + "timeBin": "DAY", + "on": false, + "match": true, + "lowFreq": false, + "bucketLimit": 450000, + "deDupe": true + } + ], + "dupe": { + "on": false, + "only": false, + "include": null, + "exclude": null, + "depth": 0, + "lowerBound": 99, + "upperBound": 100, + "approximate": 1, + "limitPerDupe": 15, + "checkHeader": true, + "filter": null, + "ignoreCase": true, + "score": 1, + "limit": 300 + }, + "profile": { + "on": true, + "only": false, + "include": null, + "exclude": null, + "shape": true, + "correlation": null, + "histogram": null, + "semantic": null, + "limit": 300, + "histogramLimit": 0, + "score": 1, + "shapeTotalScore": 0, + "shapeSensitivity": 0, + "shapeMaxPerCol": 0, + "shapeMaxColSize": 0, + "shapeGranular": null, + "behavioralDimension": "", + "behavioralDimensionGroup": "", + "behavioralValueColumn": "", + "behaviorScoreOff": false, + "behaviorLookback": 10, + "behaviorMinSupport": 4, + "profilePushDown": null, + "behaviorRowCheck": true, + "behaviorTimeCheck": true, + "behaviorMinValueCheck": true, + "behaviorMaxValueCheck": true, + "behaviorNullCheck": true, + "behaviorEmptyCheck": true, + "behaviorUniqueCheck": true, + "adaptiveTier": null + }, + "source": { + "on": false, + "only": false, + "validateValues": false, + "matches": false, + "sourcePushDownCount": false, + "include": null, + "exclude": null, + "includeSrc": null, + "excludeSrc": null, + "key": null, + "map": null, + "score": 1, + "limit": 30, + "dataset": "", + "driverName": "", + "user": "", + "password": "", + "passwordManager": "", + "connectionName": "", + "connectionUrl": "", + "query": "", + "lib": "", + "checkType": true, + "checkCase": false, + "validateValuesFilter": "", + "validateSchemaOrder": false, + "connectionProperties": {}, + "filePath": "", + "fileQuery": "", + "fullFile": false, + "header": null, + "skipLines": 0, + "inferSchema": true, + "fileType": null, + "delimiter": ",", + "fileCharSet": "UTF-8", + "avroSchema": "", + "xmlRowTag": "", + "flatten": false, + "handleMaps": false, + "handleMixedJson": false, + "multiLine": false, + "hasHeader": true + }, + "rule": { + "on": true, + "only": false, + "lib": null, + "name": "", + "absoluteScoring": false, + "ruleBreakPreviewLimit": 6 + }, + "colMatch": { + "colMatchParallelProcesses": 3, + "colMatchDurationMins": 20, + "colMatchBatchSize": 2, + "level": "exact", + "fuzzyDistance": 1, + "connectionList": [] + }, + "spark": { + "numExecutors": 3, + "driverMemory": "", + "executorMemory": "", + "executorCores": 1, + "conf": "", + "queue": "", + "master": "local[*]", + "principal": "", + "keyTab": "", + "deployMode": "", + "jars": null, + "packages": null, + "files": null + }, + "env": { + "jdbcPrincipal": "", + "jdbcKeyTab": "" + }, + "record": { + "on": false, + "in": "", + "notIn": "", + "include": null, + "percDeltaLimit": 0.1, + "score": 1 + }, + "transforms": [], + "pipeline": [] + } +``` + + + +![](<../../.gitbook/assets/image (37).png>) + +### JWT Token For Auth + +``` +import requests +import json +url = "http://localhost:9000/auth/signin" +payload = json.dumps({ + "username": "", + "password": "", + "iss": "public" +}) +headers = { + 'Content-Type': 'application/json' +} +response = requests.request("POST", url, headers=headers, data=payload) +print(response.text) +``` + +``` +curl --location --request POST 'http://localhost:9000/auth/signin' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "username": "", + "password": "", + "iss": "public" +}' +``` + +## Python Example + +Alternatively, you can use the rest endpoints directly. This example shows how it can be done with Python. + +1. Create a dataset def + 1. using the UI (Explorer) or + 2. using the dataset-def-api (https://\/swagger-ui.html#/dataset-def-api) +2. Confirm your Python environment has the appropriate modules and imports +3. Fill-in the variables and customize to your preference + 1. url, user and pass + 2. dataset, runDate, and agentName + +```python +import requests +import json + +# Authenticate +owl = "https://" +url = "https:///auth/signin" +payload = json.dumps({ + "username": "", # Edit Here + "password": "", # Edit Here + "iss": "public" # Edit Here +}) +headers = { + 'Content-Type': 'application/json' +} +response = requests.request("POST", url, headers=headers, data=payload, verify=False) +owl_header = {'Authorization': 'Bearer ' + response.json()['token']} + + +# Run +dataset = '' # Edit Here +runDate = '2021-08-08' # Edit Here +agentName = ') + +![](<../../.gitbook/assets/image (37).png>) diff --git a/apis/rest-apis/assignmentapi.md b/apis/rest-apis/assignmentapi.md new file mode 100644 index 00000000..afab1bb0 --- /dev/null +++ b/apis/rest-apis/assignmentapi.md @@ -0,0 +1,5 @@ +# Assignment API + +![](<../../.gitbook/assets/image (44) (1).png>) + +![](<../../.gitbook/assets/image (47) (1).png>) diff --git a/apis/rest-apis/export-and-import-api.md b/apis/rest-apis/export-and-import-api.md new file mode 100644 index 00000000..c538de1c --- /dev/null +++ b/apis/rest-apis/export-and-import-api.md @@ -0,0 +1,245 @@ +--- +description: Promoting and moving datasets across environments +--- + +# Export and Import API + +## Pre-requirements + +{% hint style="warning" %} +The database needs the [stored procedure](export-and-import-api.md#stored-procedure) (function) defined in order to use the Export/Import API. +{% endhint %} + +### V2 - Stable - available from 2022.02 release + +#### Step 1 - Export content from source schema + +{% swagger method="get" path="/db-export" baseUrl="https:///v2" summary="Export tables from database" %} +{% swagger-description %} + +{% endswagger-description %} + +{% swagger-parameter in="query" name="datasets" required="true" type="List of strings" %} +List of datasets to export. You need to give at least one + +**valid** + + dataset name into the list +{% endswagger-parameter %} + +{% swagger-parameter in="query" name="schema" type="String" %} +Name of the schema/tenant where you want to perform the export. + +\ + + + + +_Default value: **public**_ +{% endswagger-parameter %} + +{% swagger-parameter in="query" type="String" name="tables" %} +List of tables to export on the given schema & dataset(s). + +\ + + +If you leave it empty, the following tables will be exported altogether: + +_**rule_repo, owl_catalog, owl_rule, alert_cond, owl_check_repo, job_schedule, alert_output**_ +{% endswagger-parameter %} + +{% swagger-response status="200: OK" description="List of SQL - INSERT statements as JSON list" %} +```javascript +{ + // Response +} +``` +{% endswagger-response %} + +{% swagger-response status="400: Bad Request" description="Any error happened with error message" %} +```javascript +{ + // Response +} +``` +{% endswagger-response %} +{% endswagger %} + +#### Step 1 - Import content + +{% swagger method="post" path="/db-import" baseUrl="https:///v2" summary="Import content into the target tenant" %} +{% swagger-description %} +The target schema/tenant name will be part of the input SQL INSERT statements. + +The import is rung on non-transactional mode, any error happens in the middle, the saved items will be left in the database as is. +{% endswagger-description %} + +{% swagger-parameter in="body" type="" required="true" %} +List of SQL INSERT which will be imported into the target Collibra DQ metastore + +Format: JSON string list +{% endswagger-parameter %} + +{% swagger-response status="200: OK" description="When the import was successful" %} +```javascript +{ + // Response +} +``` +{% endswagger-response %} + +{% swagger-response status="400: Bad Request" description="Any error happened with error message" %} +```javascript +{ + // Response +} +``` +{% endswagger-response %} +{% endswagger %} + + + +### V1 - This API is going to be deprecated soon + +#### Step 1 - Get-Exports + +Best practice is to use the get-exports endpoint for most scenarios. You can pass in several dataset names and several tables at once. This endpoint will create a JSON payload + +{% hint style="info" %} +Exports and Imports are currently limited to the 3 tables listed below. +{% endhint %} + +**These are the three most common tables. These are the supported tables for re-promotion (running the export multiple times). The most common use case is to copy jobs and rules from environment A to environment B. Running the export/import sequence on the same environment likely result in a key constraint conflict, unless in-between edits are made to the insert payload.** + +* owl\_rule +* job\_schedule +* owl\_check\_repo + +``` +http:///v2/get-exports?dataset=public.dataset_scan_2,public.dataset_scan_1&schema=public&tables=owl_rule,job_schedule,owl_check_repo +``` + +#### Use Swagger to build this for you + +This is located under controller-scala (internal API) + +![](<../../.gitbook/assets/image (133).png>) + +#### Click Try it out to input the details + +![](<../../.gitbook/assets/image (58).png>) + +#### Step 2 - Run-Import + +{% hint style="info" %} +You will want to perform a find/replace on the import payload to check for differences in connections, agents, spark and environment configurations. Migrating to different environments typically requires the payload to be modified. +{% endhint %} + +Run import on the desired environment, passing the output of the previous statement to the body of the request + +```javascript +http:///v2/run-import +``` + +#### Use Swagger to try it out + +This is under controller-catalog + +![](<../../.gitbook/assets/image (125).png>) + +![](<../../.gitbook/assets/image (115).png>) + +This would be the body of the POST. + +![](<../../.gitbook/assets/Screen Shot 2021-04-26 at 10.13.18 AM.png>) + +## Requirement - Stored Procedure + +The following function needs to be created in the Collibra DQ metastore, before this can run. + +``` +CREATE OR REPLACE FUNCTION public.dump(p_schema text, p_table text, p_where text) + RETURNS SETOF text + LANGUAGE plpgsql +AS $function$ + DECLARE + dumpquery_0 text; + dumpquery_1 text; + selquery text; + selvalue text; + valrec record; + colrec record; + BEGIN + + -- ------ -- + -- GLOBAL -- + -- build base INSERT + -- build SELECT array[ ... ] + dumpquery_0 := 'INSERT INTO ' || quote_ident(p_schema) || '.' || quote_ident(p_table) || '('; + selquery := 'SELECT array['; + + <> + FOR colrec IN SELECT table_schema, table_name, column_name, data_type + FROM information_schema.columns + WHERE table_name = p_table and table_schema = p_schema + ORDER BY ordinal_position + LOOP + dumpquery_0 := dumpquery_0 || quote_ident(colrec.column_name) || ','; + selquery := selquery || 'CAST(' || quote_ident(colrec.column_name) || ' AS TEXT),'; + END LOOP label0; + + dumpquery_0 := substring(dumpquery_0 ,1,length(dumpquery_0)-1) || ')'; + dumpquery_0 := dumpquery_0 || ' VALUES ('; + selquery := substring(selquery ,1,length(selquery)-1) || '] AS MYARRAY'; + selquery := selquery || ' FROM ' ||quote_ident(p_schema)||'.'||quote_ident(p_table); + selquery := selquery || ' WHERE '||p_where; + -- GLOBAL -- + -- ------ -- + + -- ----------- -- + -- SELECT LOOP -- + -- execute SELECT built and loop on each row + <> + FOR valrec IN EXECUTE selquery + LOOP + dumpquery_1 := ''; + IF not found THEN + EXIT ; + END IF; + + -- ----------- -- + -- LOOP ARRAY (EACH FIELDS) -- + <> + FOREACH selvalue in ARRAY valrec.MYARRAY + LOOP + IF selvalue IS NULL + THEN selvalue := 'NULL'; + ELSE selvalue := quote_literal(selvalue); + END IF; + dumpquery_1 := dumpquery_1 || selvalue || ','; + END LOOP label2; + dumpquery_1 := substring(dumpquery_1 ,1,length(dumpquery_1)-1) || ');'; + -- LOOP ARRAY (EACH FIELD) -- + -- ----------- -- + + -- debug: RETURN NEXT dumpquery_0 || dumpquery_1 || ' --' || selquery; + -- debug: RETURN NEXT selquery; + RETURN NEXT dumpquery_0 || dumpquery_1; + + END LOOP label1 ; + -- SELECT LOOP -- + -- ----------- -- + + RETURN ; + END + $function$ +; +``` + +This assignment needs added. + +``` +alter function dump(text, text, text) owner to ; +``` + diff --git a/apis/rest-apis/find-the-endpoint.md b/apis/rest-apis/find-the-endpoint.md new file mode 100644 index 00000000..c23f9183 --- /dev/null +++ b/apis/rest-apis/find-the-endpoint.md @@ -0,0 +1,17 @@ +# Find the endpoint + +### Find an API Endpoint + +Any front-end action uses the API. You can find the corresponding endpoint using developer tools. + +In this example, we will look at the api call for `/create rule`. + +#### Locate the call in Developer Tools + +![](../../.gitbook/assets/docs.gif) + +#### Locate the API in Swagger + +![](../../.gitbook/assets/swagger.gif) + +All UI endpoints are the API and can be located in swagger. You can script against this externally as well. diff --git a/apis/rest-apis/owl-time-zone-api.md b/apis/rest-apis/owl-time-zone-api.md new file mode 100644 index 00000000..0aa87a0b --- /dev/null +++ b/apis/rest-apis/owl-time-zone-api.md @@ -0,0 +1,36 @@ +--- +description: UTC + global DateTime standard +--- + +# Time Zone API + +### Background on common time issues + +Controlling dates and times has always been a troublesome topic for global systems. Server clock vs server code such as new Date() which may create a date in the local timezone of the server vs the Browser or clients timezone. Moving to the cloud only makes the problem worse when you need to consider the timezone the server might be in and inherit from it's system clock. + +### Owl's Solution - Keep it Simple + +If everyone worked off of a globally understood format that is not subject to misinterpretation things would be more simple. Example: 03/02/2019 is this March second or Feb 3rd? Depends which country you live in. Owl only accepts this format: 2019-03-02. Extending this to time would mean 2019-03-02 00:00:00 + +### CmdLine Examples + +./owlcheck -ds trades -rd "2019-04-01" or -rd "2019-04-01 00:00:00" + +### Simple Example + +A user running an OwlCheck in New York and a user running an OwlCheck 3 hrs later in CA. + +| CmdLine Arg | User Location | TimeZone | Stored in Owl (UTC) | +| -------------- | ------------- | ----------- | ------------------- | +| -rd 2019-04-01 | New York | implied EST | 2019-04-01 04:00:00 | +| -rd 2019-04-01 | California | implied PST | 2019-04-01 07:00:00 | + +As you can see these jobs actually run 3 hrs apart even though they appear to run first thing in the morning to each user. Owl stores all dates in a common **UTC** format for global consistency. + +### Web API or URL Example + +[http://localhost:9000/dq/hoot?dataset=atmCustomers\&runId=2019-04-01](http://localhost:9000/dq/hoot?dataset=atmCustomers\&runId=2019-04-01) 04:00:00 [http://localhost:9000/dq/hoot?dataset=atmCustomers\&runId=2019-04-01](http://localhost:9000/dq/hoot?dataset=atmCustomers\&runId=2019-04-01) 07:00:00 + +#### For Convenience if a user prefers seeing and interacting with dates in their local time zone + +[http://localhost:9000/dq/hoot?dataset=atmCustomers\&runId=2019-04-01](http://localhost:9000/dq/hoot?dataset=atmCustomers\&runId=2019-04-01) 00:00:00\&tz=EST [http://localhost:9000/dq/hoot?dataset=atmCustomers\&runId=2019-04-01](http://localhost:9000/dq/hoot?dataset=atmCustomers\&runId=2019-04-01) 00:00:00\&tz=PST diff --git a/apis/rest-apis/rules-import-export.md b/apis/rest-apis/rules-import-export.md new file mode 100644 index 00000000..698eced4 --- /dev/null +++ b/apis/rest-apis/rules-import-export.md @@ -0,0 +1,139 @@ +# Export and Import Example + +{% hint style="info" %} +Best practice is to use get-exports and the owl\_rule table post 2021.09 release. Please refer to the [Export and Import API](export-and-import-api.md) page for more details. +{% endhint %} + +### Steps + +1. Find your dataset +2. Pass your table to the following api call - [http://\/v2/get-rules-export?dataset=public.transit\_6](http://localhost:9000/v2/get-export?dataset=public.transit\_6) +3. Run import on the desired environment, passing the output of the previous statement to the body of the request - [http://\/v2/run-import](http://35.202.14.58/v2/run-import) + +{% embed url="https://youtu.be/puXZwKi-CmM" %} + +The following function needs to be declared in the postgres metastore before this can run. + +``` +CREATE OR REPLACE FUNCTION public.dump(p_schema text, p_table text, p_where text) + RETURNS SETOF text + LANGUAGE plpgsql +AS $function$ + DECLARE + dumpquery_0 text; + dumpquery_1 text; + selquery text; + selvalue text; + valrec record; + colrec record; + BEGIN + + -- ------ -- + -- GLOBAL -- + -- build base INSERT + -- build SELECT array[ ... ] + dumpquery_0 := 'INSERT INTO ' || quote_ident(p_schema) || '.' || quote_ident(p_table) || '('; + selquery := 'SELECT array['; + + <> + FOR colrec IN SELECT table_schema, table_name, column_name, data_type + FROM information_schema.columns + WHERE table_name = p_table and table_schema = p_schema + ORDER BY ordinal_position + LOOP + dumpquery_0 := dumpquery_0 || quote_ident(colrec.column_name) || ','; + selquery := selquery || 'CAST(' || quote_ident(colrec.column_name) || ' AS TEXT),'; + END LOOP label0; + + dumpquery_0 := substring(dumpquery_0 ,1,length(dumpquery_0)-1) || ')'; + dumpquery_0 := dumpquery_0 || ' VALUES ('; + selquery := substring(selquery ,1,length(selquery)-1) || '] AS MYARRAY'; + selquery := selquery || ' FROM ' ||quote_ident(p_schema)||'.'||quote_ident(p_table); + selquery := selquery || ' WHERE '||p_where; + -- GLOBAL -- + -- ------ -- + + -- ----------- -- + -- SELECT LOOP -- + -- execute SELECT built and loop on each row + <> + FOR valrec IN EXECUTE selquery + LOOP + dumpquery_1 := ''; + IF not found THEN + EXIT ; + END IF; + + -- ----------- -- + -- LOOP ARRAY (EACH FIELDS) -- + <> + FOREACH selvalue in ARRAY valrec.MYARRAY + LOOP + IF selvalue IS NULL + THEN selvalue := 'NULL'; + ELSE selvalue := quote_literal(selvalue); + END IF; + dumpquery_1 := dumpquery_1 || selvalue || ','; + END LOOP label2; + dumpquery_1 := substring(dumpquery_1 ,1,length(dumpquery_1)-1) || ');'; + -- LOOP ARRAY (EACH FIELD) -- + -- ----------- -- + + -- debug: RETURN NEXT dumpquery_0 || dumpquery_1 || ' --' || selquery; + -- debug: RETURN NEXT selquery; + RETURN NEXT dumpquery_0 || dumpquery_1; + + END LOOP label1 ; + -- SELECT LOOP -- + -- ----------- -- + + RETURN ; + END + $function$ +; + +``` + +## From Swagger + +Navigate to the API page + + + +![](<../../.gitbook/assets/image (47).png>) + +Find the Rest APIs link + +![](<../../.gitbook/assets/image (59).png>) + +Drill-in to the controller-scala section + +![](<../../.gitbook/assets/image (49).png>) + +Find the get-rules-export call + +![](<../../.gitbook/assets/image (50).png>) + +Click Try it out and enter a dataset name, Execute to run the call + +![](<../../.gitbook/assets/image (52).png>) + +Copy the response body + +![](<../../.gitbook/assets/image (53).png>) + +Navigate to the controller-catalog section + +![](<../../.gitbook/assets/image (54).png>) + +Find run-import and Try it out + +![](<../../.gitbook/assets/image (55).png>) + +Make any edits and paste in the response body from the previous step + +![](<../../.gitbook/assets/image (56).png>) + +Visually validate the rules were transferred to another dataset successfully + +![](<../../.gitbook/assets/image (57).png>) diff --git a/apis/swagger.md b/apis/swagger.md new file mode 100644 index 00000000..9a03b1c8 --- /dev/null +++ b/apis/swagger.md @@ -0,0 +1,32 @@ +--- +description: Swagger is in the Collibra DQ application +--- + +# Swagger + +### Docs built into the application + +Collibra DQ comes with full swagger support out of the box. + +``` +http:///swagger-ui.html +``` + +Swagger can be found in the application under the Admin section labeled APIs. + +![](<../.gitbook/assets/image (120).png>) + +You will find a direct link to the Swagger page + +![](<../.gitbook/assets/image (117).png>) + +Toggle between Product API and Internal API + +![](<../.gitbook/assets/image (151).png>) + + + +For example swagger API please visit - [http://\:9000/v2/api-docs?group=UI Internal](http://35.194.91.201:9003/v2/api-docs?group=UI%20Internal) + +![](../.gitbook/assets/owl-swagger.png) + diff --git a/architecture/diagram/README.md b/architecture/diagram/README.md new file mode 100644 index 00000000..c2088564 --- /dev/null +++ b/architecture/diagram/README.md @@ -0,0 +1,35 @@ +# Architecture Diagram + +### Collibra DQ Architecture + +![](<../../.gitbook/assets/image (98).png>) + +### High-Level Diagram + +![](<../../.gitbook/assets/image (114).png>) + +### Kubernetes Diagram + +![](<../../.gitbook/assets/image (107).png>) + +### Owl Standalone + +![Owl stand alone](https://lh3.googleusercontent.com/v8y\_20neYCCo1FoOAkc3Zc\_xItIPevxxVGiNbYy5L1F9oi-lJF5lyNkAr5ZMN00ERbEU4FS\_j3kvZzgUf08ceeo0U0Sgc3tnFflOuVShrRfETUSkhtYOmPQxmaCaGpHO2hrWIW\_V) + +The image above depicts owl-web, owl-core, postgres and orient all deployed on the same server. This can be an edge node of a Hadoop cluster or a server that has access to run spark-submit jobs to the hadoop cluster. This server could also have JDBC access to other DB engines interested in being quality scanned by Owl. Looking at this depiction from left to right the client uses their browser to connect to Owl’s Web Application running on port 9000 (default port). The Owl Web Application communicates with the two metastores (Postgres and Orient). The Web Application can run a local owlcheck job, or the owlcheck script can be launched from the CLI natively. The owlcheck will launch a job using Owl’s built in Spark Local DQ Engine. Depending on the options supplied to the owlcheck command the DQ job can scan a file or database with JDBC connectivity. + +### Owl Distributed + +![Owl Semi-Distributed](https://lh4.googleusercontent.com/j4licslWJZLkm1CKVrD9P98RY8Ycjzphs1tUtVqrQuSms29yu5z4inBW3yxBCUR6JxpP8TIR45fiCshSJXVIHKQshIW4bREApPduNbewWf6cUPuvp\_26gYFcl1wruEsJEecMQrF4) + +The image above depicts owl-web and owl-core deployed on different servers. Example Owl-web would NOT be deployed on the edge node. Owl-core would be installed on the edge node and write Owlcheck results back to the metastore that Owl-web points to (NOTE in this scenario the metastore and the web-app are running on the same host). The other change is that the owlcheck job will distribute the work on top of a Hadoop cluster in order to leverage spark and use the parallel processing that comes with the Hadoop engine itself. + +### Owl Fully Distributed + +![](<../../.gitbook/assets/image (77).png>) + +The image above depicts all components deployed on different servers. OrientDB setup as a cluster which provides fault tolerance in case there is an issue with a single machine (NOTE: starting with version 1.1.0 of Owl, Orient is an optional service – or Owl-Web and Owlcheck do not rely on Orient being Available). Postgres setup in a cluster in case one server fails. Multiple owl-web deployed behind a load balancer yet communicating with the same Metadata. Owl-core would be installed on the edge node and write Owlcheck results back to the metastores that Owl-web points to. In this depiction the owlcheck driver runs on the Edge Node while the majority of the work gets pushed to the cluster (deploymode = client flag was sent), so all communication connects from the edge node to the metastores. + +![](<../../.gitbook/assets/image (78).png>) + +In this depiction the owlcheck driver and job gets pushed to the cluster (deploymode = cluster flag was sent), so all communication connects from any node on the cluster back to the metastore. diff --git a/architecture/diagram/erd.md b/architecture/diagram/erd.md new file mode 100644 index 00000000..19543984 --- /dev/null +++ b/architecture/diagram/erd.md @@ -0,0 +1,5 @@ +# ERD + +Please note there are over 110 tables in the underlying Postgres database and many are for application settings. Please contact us to receive a complete list. + +![](<../../.gitbook/assets/image (38).png>) diff --git a/architecture/system-requirements/README.md b/architecture/system-requirements/README.md new file mode 100644 index 00000000..c85b718f --- /dev/null +++ b/architecture/system-requirements/README.md @@ -0,0 +1,72 @@ +# System Requirements + +### Prerequisites + +#### Supported OS System Specification (Running owl-web and owl-core): + +Operating System (formal Collibra support only available for the following): + +* RHEL6 RHEL7 +* Centos6 Centos 7 +* Amazon Linux + +Hardware Based on Role (standalone, distributed, or fully-distributed). + +* Please see [hardware sizing](hardware-sizing.md) for minimum resources for each component. + +Java Version + +* Oracle JDK version 1.8.0\_152.x +* Open JDK version 1.8.0.x + +Encryption + +* Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction + +Client Browser + +* Chrome 70.0.3538.102 +* Firefox - 52.8.0 +* Safari Version 12.0.1 + +Postgres Version + +* Owl come prepackaged with version 11.4 of postgres +* Version 9.6.5 and above is supported if wanting to use an external metastore + +#### User Privileges: + +* Installation is completed via tarball +* User needs to be able to create directories, launch scripts, and start processes (java processes) +* SUDO is not required +* ULIMIT settings of 4096 or higher (see below) + * All owl services typically consume about 428 threads. + * During each owlcheck about 400 additional threads are consumed. + * Thus 4096 threads can allow for about 9 concurrent owlcheck jobs (on a standalone install). If more are needed plan accordingly. + +#### Default Ports used by Owl + +* 5432 - Postgres +* 9000 – Owl-web + +### Planning the installation + +It is alway best to consult the field team from Owl for more information about what options make the most sense for your environment. + +1. Determine how we are planning to install Owl (standalone or distributed) as shown in Owl-Analytics Architecture section +2. Plan infrastructure, scale, and HA +3. Validate your system prereqs +4. Obtain the Packages from Owl + +### Installation Packages/Files (BOM) + +* demoscripts.tar.gz +* log4j\* +* owlcheck +* owl-core-2.1.0-jar-with-dependencies.jar +* owl-webapp-2.1.0.jar +* owl-agent-2.1.0.jar +* setup.sh +* owl-postgres.tar.gz +* notebooks.tar.gz +* owlmanage.sh diff --git a/architecture/system-requirements/hardware-sizing.md b/architecture/system-requirements/hardware-sizing.md new file mode 100644 index 00000000..049903f9 --- /dev/null +++ b/architecture/system-requirements/hardware-sizing.md @@ -0,0 +1,55 @@ +# Hardware Sizing + +Hardware Sizing (Standalone Install) + +## Small Tier - 16 Core, 128G RAM (r5.4xlarge / E16s v3) + +| Component | RAM | Cores | +| --------- | ---- | ----- | +| Web | 2g | 2 | +| Postgres | 2g | 2 | +| Spark | 100g | 10 | +| Overhead | 10g | 2 | + +## Medium Tier - 32 Core, 256G RAM (r5.8xlarge / E32s v3) + +| Component | RAM | Cores | +| --------- | ---- | ----- | +| Web | 2g | 2 | +| Postgres | 2g | 2 | +| Spark | 250g | 26 | +| Overhead | 10g | 2 | + +## Large Tier - 64 Core, 512G RAM (r5.16xlarge / E64s v3) + +| Component | RAM | Cores | +| --------- | ---- | ----- | +| Web | 4g | 3 | +| Postgres | 4g | 3 | +| Spark | 486g | 54 | +| Overhead | 18g | 4 | + +## Estimates + +Sizing should allow headroom and based on peak concurrency and peak volume requirements. If concurrency is not a requirement, you just need to size for peak volume (largest tables). Best practice to efficiently scan is to scope the job by selecting critical columns. See [performance tuning](https://app.gitbook.com/o/-L\_xJcI2iSgW9EyVdYge/s/-Lb9zlrkw1AEC\_k2s2wG-2632701976/troubleshooting/performance-tuning) for more information. + +| Bytes per Cell | Rows | Columns | Gigabytes | Gigabytes for Spark (3x) | +| -------------- | ---------------- | ------- | --------- | ------------------------ | +| 16 | 1,000,000.00 | 25 | 0.4 | 1.2 | +| 16 | 10,000,000.00 | 25 | 4 | 12 | +| 16 | 100,000,000.00 | 25 | 40 | 120 | +| 16 | 1,000,000.00 | 50 | 0.8 | 2.4 | +| 16 | 10,000,000.00 | 50 | 8 | 24 | +| 16 | 100,000,000.00 | 50 | 80 | 240 | +| 16 | 1,000,000.00 | 100 | 1.6 | 4.8 | +| 16 | 10,000,000.00 | 100 | 16 | 48 | +| 16 | 1,000,000,000.00 | 100 | 1600 | 4800 | +| 16 | 100,000,000.00 | 100 | 160 | 480 | +| 16 | 1,000,000.00 | 200 | 3.2 | 9.6 | +| 16 | 10,000,000.00 | 200 | 32 | 96 | +| 16 | 100,000,000.00 | 200 | 320 | 960 | +| 16 | 1,000,000,000.00 | 200 | 3200 | 9600 | + +## Cluster + +If your program requires more horsepower or (spark) workers than the example tiers above which is fairly common in Fortune 500 companies than you should consider the horizontal and ephemeral scale of a cluster. Common examples are Amazon EMR, Cloudera CDP etc... OwlDQ is built to scale up horizontally and can scale to hundreds of nodes. diff --git a/benchmarks/performance-settings.md b/benchmarks/performance-settings.md new file mode 100644 index 00000000..29b9a451 --- /dev/null +++ b/benchmarks/performance-settings.md @@ -0,0 +1,17 @@ +# Performance Settings + +### Job Limits + +Limits can be set to limit resources that are requested. There are options for cores, memory, executors, and cells (maxexecutorcores, maxexecutormemory, maxnumexecutors and maxcellcountforparalleljdbc) + +![](../.gitbook/assets/limits.gif) + +If you request more cells than the limit, the user should see a warning message before hitting run. + +![](<../.gitbook/assets/image (132).png>) + +### Agent Defaults + +Set defaults at the agent level. These should be right-sized to your environment and be used as defaults for jobs with when estimate is not available (primarily local files and remote files). + +![](<../.gitbook/assets/image (144).png>) diff --git a/benchmarks/performance-tests.md b/benchmarks/performance-tests.md new file mode 100644 index 00000000..3516a3e9 --- /dev/null +++ b/benchmarks/performance-tests.md @@ -0,0 +1,77 @@ +# Performance Tests + +## Cells Per Second Performance Theory (9.5M CPS) + +![](<../.gitbook/assets/Screen Shot 2020-12-01 at 10.31.09 AM.png>) + +## Load and Profile + +|

Dataset

Name

|

GBs in

Memory

|

Rows

|

Cols

|

Cells

|

Num

Execs

|

Num

Cores

|

Exec

Memory

|

Network

Time

|

Total

Time

| +| -------------------------- | -------------------------- | ------------------ | ------------------ | ------------------- | ----------------------- | ---------------------- | ------------------------ | -------------------------- | ------------------------ | +| NYSE | 0.1G | 103K | 9 | 816K | 1 | 1 | 1G | 00:00:15 | 00:00:48 | +| AUM | 14G | 9M | 48 | 432M | 5 | 1 | 4G | 00:01:20 | 00:03:50 | +| ENERGY | 5G | 43M | 6 | 258M | 8 | 3 | 3G | 00:00:00 | 00:04:35 | +| INVEST\_DATA | 20G | 3.8M | 158 | 590M | 3 | 2 | 3G | 00:00:40 | 00:03:32 | + +### NYSE + +Postgres database call, no concurrent processing, simple case, small data. + +```bash +-bhtimeoff -numexecutors 1 +-lib "/opt/owl/drivers/postgres" +-executormemory 1g +-h metastore01.us-east1-b.c.owl-hadoop-cdh.internal:5432/dev?currentSchema=public +-drivermemory 1g -master k8s:// -ds public.nyse_128 -deploymode cluster +-q "select * from public.nyse" -bhlb 10 -rd "2020-10-26" +-driver "org.postgresql.Driver" -bhminoff +-loglevel INFO -cxn postgres-gcp -bhmaxoff +``` + +### AUM + +Postgres database call uses parallel JDBC, split on aum\_id serial id. + +```bash +-owluser kirk +-lib "/opt/owl/drivers/postgres" -datashapeoff +-numpartitions 6 -ds public.aum_dt2_50 +-deploymode cluster -bhlb 10 -bhminoff +-cxn postgres-gcp -bhmaxoff -bhtimeoff +-numexecutors 6 +-executormemory 4g -semanticoff +-h metastore01.us-east1-b.c.owl-hadoop-cdh.internal:5432/dev?currentSchema=public +-columnname aum_id -corroff -drivermemory 4g -master k8s:// +-q "select * from public.aum_dt2" -histoff -rd "2020-10-27" +-driver "org.postgresql.Driver" -loglevel INFO -agentjobid 7664 +``` + +### ENERGY + +HDFS file with 43 million rows, converting a string date to date type, deploy mode client. + +```bash +-f "hdfs:///demo/owl_usage_all.csv" \ +-rd "2019-02-02" \ +-ds energy_file \ +-loglevel DEBUG -readonly \ +-d "," -df dd-MMM-yy \ +-master yarn \ +-deploymode client \ +-numexecutors 3 \ +-executormemory 10g +``` + +## Load Profile Outliers + +### NYSE - 1:10 total runtime. 20 seconds for outliers + +```bash +-bhtimeoff -owluser kirk -numexecutors 1 +-lib "/opt/owl/drivers/postgres" -executormemory 1g +-dl -h metastore01.us-east1-b.c.owl-hadoop-cdh.internal:5432/dev?currentSchema=public +-drivermemory 1g -master k8s:// -ds public.nyse_128 -deploymode cluster +-q "select * from public.nyse" -bhlb 10 +-rd "2020-10-27" -driver "org.postgresql.Driver" +-bhminoff -loglevel INFO -cxn postgres-gcp -bhmaxoff -agentjobid 7721 +``` diff --git a/benchmarks/performance-tuning/README.md b/benchmarks/performance-tuning/README.md new file mode 100644 index 00000000..96f3bcb5 --- /dev/null +++ b/benchmarks/performance-tuning/README.md @@ -0,0 +1,130 @@ +--- +description: Guide for tuning +--- + +# Performance Tuning + +| Storage Format |

Num

Rows

|

Num

Columns

| Bytes Disk |

Num

Executors

| Executor Memory | Total RAM | Transfer Time | Process Time | +| -------------- | --------------------- | ------------------------ | ---------- | -------------------------- | --------------- | --------- | ------------- | ------------ | +| Local File | 1M | 50 | 1G | 1 | 3G | 3G | 0 mins | 2 mins | +| HDFS File | 10M | 50 | 5G | 3 | 8G | 24G | 0 mins | 4 mins | +| Hive Table | 10M | 50 | 5G | 3 | 8G | 24G | 0 mins | 4 mins | +| JDBC Table | 50M | 50 | 25G | 8 | 10G | 80G | 3 mins | 8 mins | +| JDBC Table | 10M | 100 | 10G | 3 | 12G | 36G | 3 mins | 6 mins | +| JDBC Table | 250M | 9 | 10G | 5 | 7G | 35G | 14 mins | 15 mins | +| JDBC Table | 250M | 145 | 70G | 17 | 12G | 204G | 28 mins | 30 mins | + +Using a 10/1 ratio of RAM to Executors is often a good rule of thumb, another and more simple option is to turn on dynamic.allocation and allow the resources to be provided as needed on demand. + +### Limit Columns + +In most cased there are a large number of columns that go unused by the business or columns that don't require checking. One of the most efficient things you can do is limit the cols using the below cmds. As a best practice Owl strongly recommends using less than 80 columns per dataset. + +```bash +-q "select colA, colB, colC, datCol, colD from table" +// vs +-q "select * from * from table" +``` + +#### How to limit columns when using a file + +```bash +-fq "select colA, colB, colC from dataset" +// file query using keyword dataset +``` + +## JDBC vs Local Data + +#### Co-Located data (local data) + +It is always a good performance practice to colocate data and processing. That doesn't mean that you tech organization chooses to do this in it's architecture and design which is why Owl accounts for both. If the data is located on the cluster that is doing the processing use options like -hive for non JDBC and native file access. Skip tuning for JDBC as moving data to the cluster first will routinely reduce 50% of the performance bottleneck. + +#### JDBC + +**Set fetchsize** \ +1M rows -connectionprops fetchsize=1000\ +5M rows -connectionprops fetchsize=5000\ +10M rows -connectionprops fetchsize=10000 + +**Set DriverMemory**\ +add more memory to the driver node as it will be responsible for the initial landing of data.\ + + +``` +--driver-memory 7g +``` + +**Add Parallel JDBC** + +![](../../.gitbook/assets/owl-parallel-jdbc.png) + +### Limit Features, Turn Flags Off + +``` +-corroff //only losing visuals, 5% speed gain +-histoff //only losing visuals, 4% speed gain +-hootonly //speeds up 1% based on less logging +-readonly //remove owl webapp read writes, 1% gain +-datashapeoff //removes Shape Detection 3% speed gain +``` + +### Real World Scenario + +9 Million rows with 46 columns on a daily basis for just 1 dataset. The data lives in Greenplum and we want to process it on a cluster platform where Owl runs. The first run results in a 12 minute runtime. While acceptable it's not ideal, here is what you should do. + +1. Add Parallel JDBC for faster network +2. Limit columns to the 18 that are of use in the downstream processing +3. Turn off unneeded features. +4. Find out of the job is memory bound or CPU bound + +By setting the below configs this same job ran in 6 mins. + +```bash +# parallel functions +-columnname run_date -numpartitions 4 \ +-lowerbound "2019-02-23 00:00:00" \ +-upperbound "2019-02-24 00:00:00" +# driver optimization +-connectionprops fetchsize=6000 +# analyst functions +-corroff \ +-histoff +# hardware +-executormemory 4g +-numexecutors 3 +``` + +#### The Full OwlCheck + +```bash +./owlcheck \ +-u u -p pass \ +-c jdbc:postgresql://$host/postgres \ # jdbc url +-ds aumdt -rd 2019-05-05 \ +-q "select * from aum_dt" \ +-driver org.postgresql.Driver \ # driver +-lib /home/owl/drivers/postgres \ # driver jar +-connectionprops fetchsize=6000 \ # driver performance setting +-master yarn -deploymode client \ +-executormemory 2G -numexecutors 2 -drivermemory 3g \ # hardware sizing +-h cdh-edge.us-east1-b.c.owl-hadoop-cdh.internal:2181 \ # owl metastore +-corroff -histoff -statsoff \ # owl features +-loglevel INFO \ # log level +-columnname updt_ts -numpartitions 12 \ # parallel jdbc +-lowerbound 1557623033193 -upperbound 1557623051585 +``` + +```aspnet +{ + "dataset": "aumdt", + "runId": "2019-05-05", + "score": 100, + "behaviorScore": 0, + "rows": 9000000, + "passFail": 0, + "peak": 0, + "avgRows": 0, + "cols": 46, + "runTime": "00:05:23", + } +``` diff --git a/benchmarks/performance-tuning/benchmark-faq.md b/benchmarks/performance-tuning/benchmark-faq.md new file mode 100644 index 00000000..178ebb52 --- /dev/null +++ b/benchmarks/performance-tuning/benchmark-faq.md @@ -0,0 +1,31 @@ +# Performance Considerations + +**Performance Recommendations** + +Performance is a function of available hardware. + +When running DQ Scans on a Hadoop distribution: + +* Check YARN resource manager +* Check limits on queue size +* Contact Platform Administration team on any limitations + +When running DQ Scans on Spark standalone (single node): + +* Check Spark endpoint (example: http://\:\) +* Suggested maximum size on 16 core x 64GB machine: 100 million rows \* 200 columns = 2 billion cells +* If exceeding 2 billion cells, limit the width by selecting certain columns or limit depth with a WHERE clause or a FILTER condition + +When running DQ Scans on EKS + +* Check your compute pool for available pods +* Check your worker configuration and your Spark operator configuration +* Check minimum and maximum of allowed workers + +**How-To** + +Increment DQ Scans with gradually increasing limits. Starting with a low level allows you to confirm whether database has proper indexing, skip scanning, or partitioning. Incrementing also allows you to validate security and connectivity quickly. + +* Test 1: Limit 1k rows +* Test 2: Limit 1mm rows +* Test 3: Limit 10mm rows diff --git a/catalog/business-units.md b/catalog/business-units.md new file mode 100644 index 00000000..119f070f --- /dev/null +++ b/catalog/business-units.md @@ -0,0 +1,31 @@ +# Business Units + +![](../.gitbook/assets/business\_units.gif) + +Collibra DQ allows its users to run large sets of DQ jobs. Business units provide a way to categorize and group these. + +Through the admin console the user can navigate to the business units management page. This page will contain a table of your business units and a button to create new units. + +![Business Unit Management page](<../.gitbook/assets/Screen Shot 2020-12-28 at 1.08.45 PM.png>) + +By clicking the Add Business Unit button the user will be able to fill out the business unit form. A unique name is required and a parent business unit can be optionally selected. Each unit can have one parent and many children. On the business units table you can click the blue plus icon to expand all children for that unit. If any children have children unit they will also have a plus sign to indicate the children rows can be expanded. + +![](<../.gitbook/assets/Screen Shot 2020-12-28 at 1.12.17 PM.png>) + +![](<../.gitbook/assets/Screen Shot 2020-12-28 at 1.13.35 PM.png>) + +Each business unit has actions. The business unit can be edited or deleted. If the unit is assigned to at least one data set, it cannot be deleted. If the user wishes to create children of a unit they can choose the Add Child action to have the new business unit form pre populated with the selected unit as the parent. + +![](<../.gitbook/assets/Screen Shot 2020-12-28 at 1.17.36 PM.png>) + +There are two different ways to assign a business unit to a data set. On the Catalog page there is an action option to manage the business unit. One the profile page the user can click the add/edit business units icon to open a control to manage the assigned business uint. + +![Manage business unit from Catalog](<../.gitbook/assets/Screen Shot 2020-12-28 at 1.20.48 PM.png>) + +![Manage business unit from Profile](<../.gitbook/assets/Screen Shot 2020-12-28 at 1.21.42 PM.png>) + +Once a business unit has been assigned to a data set the user can filter by that business unit on the Catalog and Pulse View. + +![Filter by business unit from Catalog](<../.gitbook/assets/Screen Shot 2020-12-28 at 1.26.01 PM.png>) + +![Filter by business unit from Pulse View](<../.gitbook/assets/Screen Shot 2020-12-28 at 1.28.45 PM.png>) diff --git a/catalog/catalog-bulk-actions.md b/catalog/catalog-bulk-actions.md new file mode 100644 index 00000000..384cc8dd --- /dev/null +++ b/catalog/catalog-bulk-actions.md @@ -0,0 +1,27 @@ +# Catalog Bulk Actions + +![](../.gitbook/assets/bulk\_actions.gif) + +### Bulk Delete + +The catalog allows user to delete multiple datasets in one action through the catalog. Click the Bulk Actions button dropdown in the top right corner. Select Bulk Delete. + +![](<../.gitbook/assets/Screen Shot 2021-04-13 at 4.23.32 PM.png>) + +Each data set row will now have a checkbox in the Actions column. All filters can be applied now and the user can check all data sets they wish to delete. Once desired datasets are selected, the checkbox can be clicked. The user will be prompted to confirm they wish to delete datasets. The user can also click the X button to cancel. + +![](<../.gitbook/assets/Screen Shot 2021-04-13 at 4.30.29 PM.png>) + +![](<../.gitbook/assets/Screen Shot 2021-04-13 at 4.30.36 PM.png>) + +### Bulk Manage Business Units + +Another bulk feature allows the user to manage the assigned business unit to multiple datasets. Click the Bulk Actions button dropdown in the top right corner. Select Bulk Manage Business Units. + +![](<../.gitbook/assets/Screen Shot 2021-04-13 at 4.34.48 PM.png>) + +Each data set row will now have a checkbox in the Actions column. All filters can be applied now and the user can check all data sets they wish to delete. Once desired datasets are selected, the user can select a business unit from the business unit dropdown control in the top right corner and click the checkbox. The user will be prompted to confirm they wish to change the assigned business unit to the datasets. + +![](<../.gitbook/assets/Screen Shot 2021-04-13 at 4.38.15 PM.png>) + +![](<../.gitbook/assets/Screen Shot 2021-04-13 at 4.38.21 PM.png>) diff --git a/catalog/catalog.md b/catalog/catalog.md new file mode 100644 index 00000000..391a2179 --- /dev/null +++ b/catalog/catalog.md @@ -0,0 +1,21 @@ +# Overview + +### Smart Catalog - Bringing Data Science to Cataloging + +While Owl does not pride itself on being a catalog tool it does automatically maintain a dataset and process catalog. It is a necessary control for Owl and helpful to the end user. Without a smart catalog a user could technically overwrite another user's OwlCheck (DQ check). For example -ds "Trade" and -ds "Trade". Owl believes a healthier habit is to store the full natural name of the dataset and allow the user to alias the name in the event that they wish to make a short-name. By doing this Owl can protect users from mixing up their results and stops the constant renaming of common objects which leads to more unnecessary business level mapping. This approach makes it effortless for a user to create a new OwlCheck in the wizard because Owl will warn the user if there is a naming collision. Owl learns all the server hosts, the database schemas and table names and keeps things automatically organized. One less catalog to setup, manage and eventually untangle. + +### Automatic Sensitive Data (PII) Detection + +Owl automatically understands the _semantic_ schema of your data such as CREDIT CARD, EMAIL, SSN and much more. Additionally Owl will label sensitive data with PII and MNPI classifications. + +![](<../.gitbook/assets/owl-catalog-pii (2).png>) + +### Data Table View of PII + +Owl applies many labels to the header of a field / column. These labels be seen in the data preview table with highlighted errors and findings. + +![](../.gitbook/assets/owl-pii.png) + +#### The catalog offers a global view and filtering to see where PII exists + +![ You can also view globally via the catalog to see PII exists](<../.gitbook/assets/Screen Shot 2019-11-04 at 2.10.11 PM (1).png>) diff --git a/connecting-to-dbs-in-owl-web/add-connection-to-agent.md b/connecting-to-dbs-in-owl-web/add-connection-to-agent.md new file mode 100644 index 00000000..7bc7f55e --- /dev/null +++ b/connecting-to-dbs-in-owl-web/add-connection-to-agent.md @@ -0,0 +1,15 @@ +# Add Connection to Agent + +![](../.gitbook/assets/agent\_shuffle.gif) + +## How To Link DB Connection to Agent via UI + +When you add a new [Database Connection](https://docs.owl-analytics.com/connecting-to-dbs-in-owl-web/owl-db-connection#how-to-add-db-connection-via-ui), the DQ Agent must be given the permission to run DQ Job via the specified agent. + +From Fig 3, select the chain link icon next to the DQ Agent to establish link to DB Connection. A modal to add give that agent permission to run DQ Jobs by DB Connection name will show (Fig 5). The left-side panel is the list DB Connection names that has not been linked to the DQ Agent. The right-side panel is the list of DB Connection names that has the permission to run DQ Job. + +Double click the DQ Connection name to move from left to right. In Fig 5, DB Connection named "metastore" is being added to DQ Agent. Click the "Update" button to save the new list of DB Connections.Fig 5: Adding DB Connection named "metastore" to the DQ AgentFig 6: How to add all connections to the selected DQ Agent + +![](https://gblobscdn.gitbook.com/assets%2F-Lb9zlrkw1AEC\_k2s2wG%2F-MezgsSFv-xynJD2XJTM%2F-Mezhb0qV-4VR3bXoPzC%2FDQ-Agent-Config-Hint-1.png?alt=media\&token=5e85d0cb-7572-4a05-a95d-c66d5606e641) + +![](https://gblobscdn.gitbook.com/assets%2F-Lb9zlrkw1AEC\_k2s2wG%2F-McBIJaM6HGPXZ7Saycu%2F-McBP5WB-FQTCxMHwSMK%2FScreenshot%202021-06-14%20at%205.04.25%20PM.png?alt=media\&token=beca7cf8-3a72-4f64-b991-b258dd88193b) diff --git a/connecting-to-dbs-in-owl-web/owl-db-connection.md b/connecting-to-dbs-in-owl-web/owl-db-connection.md new file mode 100644 index 00000000..0ce394e8 --- /dev/null +++ b/connecting-to-dbs-in-owl-web/owl-db-connection.md @@ -0,0 +1,44 @@ +# Add a Connection + +![](<../.gitbook/assets/connections (1).gif>) + +## How to Add DB Connection via UI + +We will add a connection named `metastore` that connects to local Postgres server (`localhost:5432/postgres`) + +* Login to DQ Web and navigate to Admin Console. + +![Fig 1: Home Page](../.gitbook/assets/DQ-Admin-Console-1.png) + +* From the Admin Console, click on the Connections tile. + +![Fig 2: Admin Console](../.gitbook/assets/DQ-Admin-Console-2.png.png) + +* Click on Add button in Postgres box to add a Postgres connection + +![Fig 3: List of DB Connections](../.gitbook/assets/DQ-Connection-1.png) + +Default Postgres JDBC template connection is shown. This modal is populated with basic values what Postgres connection setting should look like. + +![Fig 4: Template Postgres connection creation modal](../.gitbook/assets/DQ-Connection-2.png) + +Replace the Connection URL to point to the Postgres server you want to run DQ Jobs against. In this example, `jdbc:postgresql://localhost:5432/postgres` + +Also change Driver Location to the JDBC Driver for Postgres in your installation. Click on the folder icon and click on Postgres driver path. These Driver Directories are default JDBC Drivers provided by DQ installation (usually in `$OWL_BASE/owl/drivers/*`) + +![Fig 5: Add new driver or select existing from Driver Directories](../.gitbook/assets/DQ-Connection-3.png) + +Fig 6 is what the new connection setting should look like. Make sure to provide the correct Postgres Username and Password (if using Username/Password for authentication). Press Save to continue. _**This action will attempt to establish a connection.**_ + +![Fig 6: Connection settings to connect to database named "postgres" in Postgres server "localhost" exposed via port 5432](../.gitbook/assets/DQ-Connection-4.png) + +### Link Connection to Agent + +{% hint style="info" %} +Make sure to [Link DB Connection](https://docs.owl-analytics.com/installation/agent-configuration#how-to-link-db-connection-to-agent-via-ui) to a DQ Agent, if required +{% endhint %} + + + + + diff --git a/connecting-to-dbs-in-owl-web/supported-drivers/README.md b/connecting-to-dbs-in-owl-web/supported-drivers/README.md new file mode 100644 index 00000000..419ee484 --- /dev/null +++ b/connecting-to-dbs-in-owl-web/supported-drivers/README.md @@ -0,0 +1,107 @@ +--- +description: A list of supported data source connection types. +--- + +# Supported Connections + +## Production + +The following is a list of drivers certified for production use. + +_Note: Illustrative grades provided below represent level of support around data types, authentication methods, and features. Example scenarios as follows:_ + +* _Data Types_ + * _A Grade: Majority supported with fewer exceptions_ + * _B- Grade: Modest support with more exceptions e.g. Boolean, Geography, Advanced Date/Time_ +* _Authentication_ + * _A Grade: Multiple methods including Kerberos keytabs_ + * _B Grade: User/pass only_ +* _Features_ + * _A Grade: Pushdown Support, Views, Special characters, Validate Source, versatile Driver support_ + * _C Grade: No pushdown support, no special characters, specific drivers required (e.g. multiple JARs)_ + +| Connection Type | Driver | Certification | Grade | Auth Type | Comments | +| ----------------- | ------------------------------------- | ------------- | ----- | ------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| **Teradata** | Native | Production | B+ | User / Pass | Some sql nuances, fairly easy to use and performs well | +| **Oracle** | Native | Production | A- | User / Pass | Performance based on how the DB is configured and fetch size in driver. | +| **MS SQL Server** | Native | Production | A- | User / Pass | allows whitespace in column headers | +| **Snowflake** | Native | Production | A+ | User / Pass | easy to use and performs well | +| **S3** | S3 SDK in Web / S3a connector in Core | Production | A | Secret / Key, Instance Profile, Assume Role | Supports bucket keys, IAM roles and caching | +| **Hive** | Simba JDBC | Production | B+ | User / Pass, Kerberos | Too many variations of Tez, mapReduce and other nuances that make it difficult to get started. | +| **Impala** | Simba JDBC | Production | A- | User / Pass | Tends to be a quicker start than Hive but still has many config nuances | +| **Postgres** | Native | Production | A+ | User / Pass | easy to use performs well | +| **MySql** | Native | Production | A+ | User / Pass | easy to use performs well | +| **DB2** | Native | Production | A- | User / Pass, Kerberos | easy to use performs well, fetch syntax vs limit and other nuances | +| **GreenPlum** | Postgres | Production | A- | User / Pass | easy to use performs well | +| **Redshift** | Simba JDBC | Production | B+ | | | +| **Athena** | Simba JDBC | Production | B+ | | | +| **Presto** | Simba JDBC | Production | B+ | | | +| **HDFS** | HDFS connector | Production | B | Kerberos | works well but usually a few hadoop spark nuances to get right | + +## Preview + +The following is a list of drivers which are for test purposes (not certified yet for production usage). + +| Connection Type | Driver | Certification | Grade | Auth Type | Comments | +| -------------------------------- | --------------------------------------------------------------------------------- | ------------- | ----- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **MongoDB** | unityJDBC | Preview | B | User / Pass | Depends which driver you use to turn a document store into a relational view. | +| **MapR Hive** | MapR Hive Driver | Preview | C+ | User / Pass | End of Life | +| **BigQuery** |

Simba JDBC (Web)

Spark Connector (Core)

| Preview | B+ | JSON service account |

Views are now supported with viewEnabled=true flag, Joins work but are not supported.

Requires 20 jars as compared to 1 jar and accepts pushdown but has nuances.

| +| **GCS (Google Cloud Storage)** |

Google Cloud SDK for Web /

GCS Spark Connector in Core

| Preview | B | JSON service account | more config than usual | +| **Azure Cloud Storage (ADLSv2)** |

Azure SDK for Web /

Azure Data Explorer connector for Spark in Core

| Preview | B- | Key / Service Principal | more config than avg | +| **Solr** | Solr JDBC | Preview | B- | User / Pass | not a relational store so requires some understanding of Solr | +| **Phoenix** | Native | Preview | C+ | | requires knowledge of phoenix and hbase | +| **Cassandra** | Native | Preview | C+ | User / Pass | CQL vs SQL and other nuances | +| **MS SQL Data Warehouse** | Native | Preview | B+ | User / Pass | | +| **Delta Lake** | Native | Preview | B- | User / Pass | Requires knowledge of databricks | +| **SAP HANA** | Native | Preview | B+ | User / Pass | works with most common data types | +| **MariaDB** | MySQL Driver | Preview | B+ | User / Pass | Uses mysql driver, some nuances | +| **Dremio** | Dremio JDBC | Preview | B | User / Pass | | +| **Kafka** | Native | Preview | B- | | Most cases the group doesn't know enough about kafka administration, schema registry and other nuances. | +| **Sybase** | Native | Preview | B+ | | | + +## Files + +| File Type Support | Grade | Comments | +| ---------------------------- | ----- | ------------------------------------------ | +| **Parquet** | B+ | schema in file, less common but works well | +| **CSV** (and all delimiters) | A+ | very common easy to use | +| **JSON** | A- | works great depends how many levels nested | +| **XML** | B | recommend json | +| **AVRO** | B | | + +### **Limitations** + +**Authentication** + +* DQ Jobs that require Kerberos TGT are not yet supported on Spark Standalone or Local deployments + * Recommended to submit jobs via Yarn or K8s + +### **File Limitations** + +**File Sizes** + +* Files with more than 250 columns supported in File Explorer, unless you have Livy enabled. +* Files larger than 5gb are not supported in File Explorer, unless you have Livy enabled. +* Smaller file sizes will allow for skip scanning and more efficient processing +* Advanced features like replay, scheduling, and historical lookbacks require a date signature in the folder of file path + +**S3** + +* Please ensure no spaces in S3 connection name +* Please remember to select 'Save Credentials' checkbox upon establishing connection +* Please point to **root** bucket, not sub folders + +**Local Files** + +* Local files can only be run using NO\_AGENT default +* This is for quick testing, smaller files, and demonstration purposes. +* Local file scanning is not intended for large scale production use. + +**Livy** + +* Livy is only supported for K8s environments + +#### Spark Engine Support + +* MapR is EOL and MapR spark engine not supported to run CDQ jobs. diff --git a/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-athena.md b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-athena.md new file mode 100644 index 00000000..85a95eab --- /dev/null +++ b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-athena.md @@ -0,0 +1,68 @@ +# Connectivity to Athena + +Your host can connect to Athena with either an Athena public service endpoint or an Athena private endpoint. For more information on setting the endpoint, see [Command line options](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-options.html) and [Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html). + +### JDBC URL Example + +jdbc:awsathena://AwsRegion=us-east-1;User=xxx;Password=xxx;S3OutputLocation=s3://data-bucket;**MetadataRetrievalMethod=Query** + +* Athena uses port 443 to connect to the host. +* Athena's streaming API uses port 444 to stream the query results. When you use a JDBC/ODBC driver, Athena uses this port to stream the query results to the JDBC/ODBC driver installed on the client host. Therefore, unblock this port when you use a JDBC/ODBC driver to connect to Athena. If this port is blocked, your business intelligence tool might time out or fail to show query results when you run a query. +* Use the appropriate JDBC connection URLs in your business tool configuration according to your private DNS configuration for your endpoint. + * Use the following connection string if you turned off the private DNS: jdbc:awsathena://vpce-.[athena.us-east-1.vpce.amazonaws.com:443](http://athena.us-east-1.vpce.amazonaws.com:443) + * Use the following connection string if you turned on the private DNS: jdbc:awsathena://[athena.us-east-1.amazonaws.com:443](http://athena.us-east-1.amazonaws.com:443) +* Be sure that the[ security group](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-access.html#vpc-endpoints-security-groups) attached to your VPC endpoint allows traffic from the host where you installed the JDBC/ODBC driver. +* Be sure that port 444 isn't blocked. If you use an AWS PrivateLink endpoint to connect to Athena, then be sure that the security group attached to the AWS PrivateLink endpoint is open to inbound traffic on port 444. Athena uses port 444 to stream query results. If port 444 is blocked, then the results aren't streamed back to your client host. In such situations, you might receive an error message similar to "\[Simba]\[AthenaJDBC]\(100123) An error has occurred. Exception during column initialization". This can also cause the business intelligence tool to stop responding and not display the query results. + +``` +telnet athena.us-east-1.amazonaws.com 443 +telnet glue.us-east-1.amazonaws.com 443 +``` + +### Minimum Permissions + +``` +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "athena:StartQueryExecution", + "s3:ListBucketMultipartUploads", + "athena:GetQueryResultsStream", + "glue:GetTables", + "glue:GetPartitions", + "athena:GetQueryResults", + "glue:BatchGetPartition", + "s3:ListBucket", + "glue:GetDatabases", + "athena:ListQueryExecutions", + "s3:ListMultipartUploadParts", + "glue:GetTable", + "glue:GetDatabase", + "athena:GetWorkGroup", + "s3:PutObject", + "s3:GetObject", + "glue:GetPartition", + "glue:GetCatalogImportStatus", + "athena:StopQueryExecution", + "athena:GetQueryExecution", + "s3:GetBucketLocation", + "athena:BatchGetQueryExecution", + "athena:DeletePreparedStatement", + "athena:CreatePreparedStatement" + ], + "Resource": [ + "arn:aws:athena:*::workgroup/primary", + "arn:aws:s3:::/*", + "arn:aws:s3:::", + "arn:aws:glue:*::catalog", + "arn:aws:glue:*::database/", + "arn:aws:glue:*::table//*" + ] + } + ] +} +``` diff --git a/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-bigquery.md b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-bigquery.md new file mode 100644 index 00000000..ce9d91d5 --- /dev/null +++ b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-bigquery.md @@ -0,0 +1,128 @@ +# Connectivity to BigQuery + +### Steps for the BigQuery Connection + +1. **Driver**: com.simba.googlebigquery.jdbc42.Driver +2. **Locate your service account owl-gcp.json** (your org auth key in JSON format) +3. **Create a JDBC connection** (for example only do not use this JDBC URL):\ + jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=;OAuthType=0;OAuthServiceAcctEmail=<1234567890>[-compute@developer.gserviceaccount.com;OAuthPvtKeyPath=/opt/ext/bq-gcp.json;Timeout=86400](mailto:-compute@developer.gserviceaccount.com;OAuthPvtKeyPath=/opt/ext/owl-gcp.json;Timeout=86400) +4. **Requires a path to a JSON file** that contains the service account for authorization. That same file is provided to the Spark session to make a direct to storage connection for maximum parallelism once Core fires up.” + +The above and explained there are actually a number of others steps which must be performed to achieve success: + +1. **Password for the BigQuery Connector form in Collibra DQ must be a base64 encoded string created from the json file (see step 3. above)** and input as password. For example:\ + `base64 your_json.json -w 0`\ + or\ + `cat your_json.json | base64 -w 0` +2. **Check that this JAR exists and is on the path of the Collibra DQ Web UI server** (eg. \/owl/drivers/bigquery/core). Look at your driver directory location which contains this BigQuery JAR: spark-bigquery\_2.12-0.18.1.jar +3. **Make sure these JARs present in \/owl/drivers/bigquery/:**\ + ****_animal-sniffer-annotations-1.19.jar_\ + _google-api-services-bigquery-v2-rev20201030-1.30.10.jar_\ + _grpc-google-cloud-bigquerystorage-v1beta1-0.106.4.jar_\ + _listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar_\ + _annotations-4.1.1.4.jar_\ + _google-auth-library-credentials-0.22.0.jar_\ + _grpc-google-cloud-bigquerystorage-v1beta2-0.106.4.jar_\ + _opencensus-api-0.24.0.jar_\ + _api-common-1.10.1.jar_\ + _google-auth-library-oauth2-http-0.22.0.jar_\ + _grpc-grpclb-1.33.1.jar_\ + _opencensus-contrib-http-util-0.24.0.jar_\ + _auto-value-annotations-1.7.4.jar_\ + _GoogleBigQueryJDBC42.jar_\ + _grpc-netty-shaded-1.33.1.jar_\ + _perfmark-api-0.19.0.jar_\ + _avro-1.10.0.jar_\ + _google-cloud-bigquery-1.125.0.jar_\ + _grpc-protobuf-1.33.1.jar_\ + _protobuf-java-3.13.0.jar_\ + _checker-compat-qual-2.5.5.jar_\ + _google-cloud-bigquerystorage-1.6.4.jar_\ + _grpc-protobuf-lite-1.33.1.jar_\ + _protobuf-java-util-3.13.0.jar_\ + _commons-codec-1.11.jar_\ + _google-cloud-core-1.93.10.jar_\ + _grpc-stub-1.33.1.jar_\ + _proto-google-cloud-bigquerystorage-v1-1.6.4.jar_\ + _commons-compress-1.20.jar_\ + _google-cloud-core-http-1.93.10.jar_\ + _gson-2.8.6.jar_\ + _proto-google-cloud-bigquerystorage-v1alpha2-0.106.4.jar_\ + _commons-lang3-3.5.jar_\ + _google-http-client-1.38.0.jar_\ + _guava-23.0.jar_\ + _proto-google-cloud-bigquerystorage-v1beta1-0.106.4.jar_\ + _commons-logging-1.2.jar_\ + _google-http-client-apache-v2-1.38.0.jar_\ + _httpclient-4.5.13.jar_\ + _proto-google-cloud-bigquerystorage-v1beta2-0.106.4.jar_\ + _conscrypt-openjdk-uber-2.5.1.jar_\ + _google-http-client-appengine-1.38.0.jar_\ + _httpcore-4.4.13.jar_\ + _proto-google-common-protos-2.0.1.jar_\ + _core_\ + _google-http-client-jackson2-1.38.0.jar_\ + _j2objc-annotations-1.3.jar_\ + _proto-google-iam-v1-1.0.3.jar_\ + _error\_prone\_annotations-2.4.0.jar_\ + _google-oauth-client-1.31.1.jar_\ + _jackson-annotations-2.11.0.jar_\ + _grpc-alts-1.33.1.jar_\ + _jackson-core-2.11.3.jar_\ + _slf4j-api-1.7.30.jar_\ + _failureaccess-1.0.1.jar_\ + _grpc-api-1.33.1.jar_\ + _jackson-databind-2.11.0.jar_\ + _gax-1.60.0.jar_\ + _grpc-auth-1.33.1.jar_\ + _javax.annotation-api-1.3.2.jar_\ + _threetenbp-1.5.0.jar_\ + _gax-grpc-1.60.0.jar_\ + _grpc-context-1.33.1.jar_\ + _joda-time-2.10.1.jar_\ + _gax-httpjson-0.77.0.jar_\ + _grpc-core-1.33.1.jar_\ + _json-20200518.jar_\ + _google-api-client-1.31.1.jar_\ + _grpc-google-cloud-bigquerystorage-v1-1.6.4.jar_\ + _jsr305-3.0.2.jar_ +4. You may get a CLASSPATH conflict regarding the JAR files. +5. Make sure the BigQuery connector Scala version matches your Spark Scala version.[![02%20PM](https://discourse-static.influitive.net/uploads/db\_033c9cc6\_3cea\_4623\_b4a8\_52ebc3f9e8a1/optimized/2X/d/dfca73373275afb5f063f192a3aa7105caa76bd8\_2\_286x500.png)](https://discourse-static.influitive.net/uploads/db\_033c9cc6\_3cea\_4623\_b4a8\_52ebc3f9e8a1/original/2X/d/dfca73373275afb5f063f192a3aa7105caa76bd8.png) + +### Networking + +Please account for these urls from a networking and firewall perspective. + +logging.googleapis.com + +oauth2.googleapis.com + +www.googleapis.com + +bigquerystorage.googleapis.com + +bigquery.googleapis.com + +### Permissions + +Make sure the project and account have appropriate permissions. These are common permissions to provide to the account. + +![](<../../.gitbook/assets/image (76).png>) + +### Views + +Support for Bigquery views is available from the 2021.11 release and forward. There are Bigquery limitations on creating views from different datasets (collections). Optionally you can add the `viewsEnabled=true` parameter to the connection property when defining the connection. + +![](<../../.gitbook/assets/image (146).png>) + +### Spark Version 2 + +{% hint style="warning" %} +Be sure to use the Spark BigQuery connector that is compatible with your version of Spark. + +Also when using Spark < 3 and Scala 2.11 please add the following props to the connection properties: +{% endhint %} + +``` +dq.bq.legacy=true,viewsEnabled=true +``` diff --git a/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-hive/README.md b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-hive/README.md new file mode 100644 index 00000000..0635c77c --- /dev/null +++ b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-hive/README.md @@ -0,0 +1,69 @@ +# Connectivity to Hive + +### Example URL + +``` +jdbc:hive2://cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal:10000/default;AuthMech=1;KrbHostFQDN=cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal;KrbRealm=CW.COM;KrbServiceName=hive;SSL=1;SSLKeyStore=/opt/cloudera/security/pki/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal-server.jks;AllowSelfSignedCerts=1;SSLKeyStorePwd=password;principal=hive/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal@CW.COM +``` + +### Driver Name + +``` +com.simba.hive.jdbc41.HS2Driver +``` + +### Driver Properties + +``` +hive.resultset.use.unique.column.names=false +``` + +### What Does each option mean + +**Base connection string** **=** jdbc:hive2://cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal:10000/default + +**Authentication identifier** **=** AuthMech=1 (which states Kerberos) + +**Kerberos Hive Server FQDN** **=** KrbHostFQDN=cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal + +**Kerberos Realm used =** KrbRealm=CW.COM _(Not necessarily needed)_ + +**Kerberos Service name =** KrbServiceName=hive + +**Enabling SSL =** SSL=1 + +**The SSL KeyStore to be used to =** SSLKeyStore=/opt/cloudera/security/pki/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal-server.jks _(Could use SSLTrustStore also)_ + +**Allow for Self Signed certifications to be OK =** AllowSelfSignedCerts=1 _(our environment used self signed certs)_ + +**Password to the KeyStore =** SSLKeyStorePwd=password _(Not necessarily needed)_ + +**Kerberos Principal to use to Authenticate with =** principal=hive/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal@CW.COM + +### Example Connection + +![](<../../../.gitbook/assets/image (75).png>) + +### Within the Owl Web UI you have to specify the following (See ScreenShot below) + +![HS2Driver Connection for Owl using HS2 SSL/TLS/Kerberos](http://18.204.201.140:8080/xwiki/bin/download/Internal%20Documentation/Enabling%20TLS%2FSSL%20for%20Hiveserver2/WebHome/Screen%20Shot%202019-06-12%20at%209.25.13%20AM.png) + +**Name =** hivessl + +**Connection URL =** jdbc:hive2://cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal:10000/default;AuthMech=1;KrbHostFQDN=cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal;KrbRealm=CW.COM;KrbServiceName=hive;SSL=1;SSLKeyStore=/opt/cloudera/security/pki/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal-server.jks;AllowSelfSignedCerts=1;SSLKeyStorePwd=password;principal=hive/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal@CW.COM + +**Port =** 10000 + +**Driver Name =** com.cloudera.hive.jdbc4.HS2Driver + +**Username =** [userspark@CW.COM](mailto:userspark@CW.COM) + +**Password =** password + +### FAQ + +It is very common to require this connection property when not using the default schema. Remember to add this to the connection properties when defining the connection. + +``` +hive.resultset.use.unique.column.names=false +``` diff --git a/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-hive/connecting-to-cdh-5.16-hive-ssl-tls-kerberos-setup.md b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-hive/connecting-to-cdh-5.16-hive-ssl-tls-kerberos-setup.md new file mode 100644 index 00000000..9862a454 --- /dev/null +++ b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-hive/connecting-to-cdh-5.16-hive-ssl-tls-kerberos-setup.md @@ -0,0 +1,47 @@ +# Connecting to CDH 5.16 Hive SSL/TLS/Kerberos Setup + +The Cloudera Hive JDBC drivers used [https://www.cloudera.com/downloads/connectors/hive/jdbc/2-5-16.html](https://www.cloudera.com/downloads/connectors/hive/jdbc/2-5-16.html) + +### JDBC Connection String used + +jdbc:hive2://cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal:10000/default;AuthMech=1;KrbHostFQDN=cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal;KrbRealm=CW.COM;KrbServiceName=hive;SSL=1;SSLKeyStore=/opt/cloudera/security/pki/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal-server.jks;AllowSelfSignedCerts=1;SSLKeyStorePwd=password;principal=hive/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal@CW.COM + +### What Does each option mean + +**Base connection string** **=** jdbc:hive2://cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal:10000/default + +**Authentication identifier** **=** AuthMech=1 (which states Kerberos) + +**Kerberos Hive Server FQDN** **=** KrbHostFQDN=cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal + +**Kerberos Realm used =** KrbRealm=CW.COM _(Not necessarily needed)_ + +**Kerberos Service name =** KrbServiceName=hive + +**Enabling SSL =** SSL=1 + +**The SSL KeyStore to be used to =** SSLKeyStore=/opt/cloudera/security/pki/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal-server.jks _(Could use SSLTrustStore also)_ + +**Allow for Self Signed certifications to be OK =** AllowSelfSignedCerts=1 _(our environment used self signed certs)_ + +**Password to the KeyStore =** SSLKeyStorePwd=password _(Not necessarily needed)_ + +**Kerberos Principal to use to Authenticate with =** principal=hive/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal@CW.COM + +#### + +### Within the Owl Web UI you have to specify the following (See ScreenShot below) + +![HS2Driver Connection for Owl using HS2 SSL/TLS/Kerberos](http://18.204.201.140:8080/xwiki/bin/download/Internal%20Documentation/Enabling%20TLS%2FSSL%20for%20Hiveserver2/WebHome/Screen%20Shot%202019-06-12%20at%209.25.13%20AM.png) + +**Name =** hivessl + +**Connection URL =** jdbc:hive2://cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal:10000/default;AuthMech=1;KrbHostFQDN=cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal;KrbRealm=CW.COM;KrbServiceName=hive;SSL=1;SSLKeyStore=/opt/cloudera/security/pki/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal-server.jks;AllowSelfSignedCerts=1;SSLKeyStorePwd=password;principal=hive/cdh-instance1.us-east1-b.c.owl-hadoop-cdh.internal@CW.COM + +**Port =** 10000 + +**Driver Name =** com.cloudera.hive.jdbc4.HS2Driver + +**Username =** [userspark@CW.COM](mailto:userspark@CW.COM) + +**Password =** password\ diff --git a/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-oracle.md b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-oracle.md new file mode 100644 index 00000000..117421cf --- /dev/null +++ b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-oracle.md @@ -0,0 +1,33 @@ +# Connectivity to Oracle + +### Example URL + +``` +jdbc:oracle:thin:@10.589.0.31:1521:DEV +``` + +### Driver Name + +``` +oracle.jdbc.OracleDriver +``` + +**Connection Properties Recognized by Oracle JDBC Drivers**\ + + +| **Name** | **Short Name** | **Type** | **Description** | +| --------------------------- | -------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| user | n/a | String | the user name for logging into the database | +| password | n/a | String | the password for logging into the database | +| database | server | String | the connect string for the database | +| internal\_logon | n/a | String | a role, such as `sysdba` or `sysoper`, that allows you to log on as `sys` | +| defaultRowPrefetch | prefetch | String (containing integer value) | the default number of rows to prefetch from the server (default value is "10") | +| remarksReporting | remarks | String (containing boolean value) | "true" if getTables() and getColumns() should report TABLE\_REMARKS; equivalent to using setRemarksReporting() (default value is "false") | +| defaultBatchValue | batchvalue | String (containing integer value) | the default batch value that triggers an execution request (default value is "10") | +| includeSynonyms | synonyms | String (containing boolean value) | "true" to include column information from predefined "synonym" SQL entities when you execute a `DataBaseMetaData getColumns()` call; equivalent to connection `setIncludeSynonyms()` call (default value is "false") | +| processEscapes | n/a | String (containing boolean value) | "false" to disable escape processing for statements (Statement or PreparedStatement) created from this connection. Set this to "false" if you want to avoid many calls to `Statement.setEscapeProcessing(false);`. This is espcially usefull for PreparedStatement where a call to `setEscapeProcessing(false)` would have no effect. The default is "true". | +| defaultNChar | n/a | String (containing boolean value) |

"false" is the default. If set to "true", the default behavior for handling character datatypes is changed so that NCHAR/NVARCHAR2 become the default. This means that setFormOfUse() won't be needed anymore when using NCHAR/NVARCHAR2. This can also be set as a java property :
java -Doracle.jdbc.defaultNChar=true myApplication

| +| useFetchSizeWithLongColumn | n/a | String (containing boolean value) |

"false" is the default.
THIS IS A THIN ONLY PROPERTY. IT SHOULD NOT BE USED WITH ANY OTHER DRIVERS.
If set to "true", the performance when retrieving data in a 'SELECT' will be improved but the default behavior for handling LONG columns will be changed to fetch multiple rows (prefetch size). It means that enough memory will be allocated to read this data. So if you want to use this property, make sure that the LONG columns you are retrieving are not too big or you may run out of memory. This property can also be set as a java property :
java -Doracle.jdbc.useFetchSizeWithLongColumn=true myApplication

| +| SetFloatAndDoubleUseBinary | n/a | String (containing boolean value) |

"false" is the default.
If set to "true", causes the java.sql.PreparedStatment setFloat and setDouble API's to use internal binary format as for BINARY_FLOAT and BINARY_DOUBLE parameters.
See oracle.jdbc.OraclePreparedStatement setBinaryFloat and setBinaryDouble

| + +[https://docs.oracle.com/cd/E11882\_01/appdev.112/e13995/oracle/jdbc/OracleDriver.html](https://docs.oracle.com/cd/E11882\_01/appdev.112/e13995/oracle/jdbc/OracleDriver.html) diff --git a/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-presto.md b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-presto.md new file mode 100644 index 00000000..4281f018 --- /dev/null +++ b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-presto.md @@ -0,0 +1,42 @@ +# Connectivity to Presto + +### Example URL + +``` +jdbc:presto://xyz.presto.svc.cluster.local:8080/database +``` + +### Driver Name + +``` +com.facebook.presto.jdbc.PrestoDriver +``` + +### Connection Properties + +These methods may be mixed; some parameters may be specified in the URL while others are specified using properties. However, the same parameter may not be specified using both methods. + +### Parameter Reference + +| Name | Description | +| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `user` | Username to use for authentication and authorization. | +| `password` | Password to use for LDAP authentication. | +| `socksProxy` | SOCKS proxy host and port. Example: `localhost:1080` | +| `httpProxy` | HTTP proxy host and port. Example: `localhost:8888` | +| `protocols` | Comma delineated list of HTTP protocols to use. Example: `protocols=http11`. Acceptable values: `http11,http10,http2` | +| `applicationNamePrefix` | Prefix to append to any specified `ApplicationName` client info property, which is used to set the source name for the Presto query. If neither this property nor `ApplicationName` are set, the source for the query will be `presto-jdbc`. | +| `accessToken` | Access token for token based authentication. | +| `SSL` | Use HTTPS for connections | +| `SSLKeyStorePath` | The location of the Java KeyStore file that contains the certificate and private key to use for authentication. | +| `SSLKeyStorePassword` | The password for the KeyStore. | +| `SSLTrustStorePath` | The location of the Java TrustStore file that will be used to validate HTTPS server certificates. | +| `SSLTrustStorePassword` | The password for the TrustStore. | +| `KerberosRemoteServiceName` | Presto coordinator Kerberos service name. This parameter is required for Kerberos authentication. | +| `KerberosPrincipal` | The principal to use when authenticating to the Presto coordinator. | +| `KerberosUseCanonicalHostname` | Use the canonical hostname of the Presto coordinator for the Kerberos service principal by first resolving the hostname to an IP address and then doing a reverse DNS lookup for that IP address. This is enabled by default. | +| `KerberosConfigPath` | Kerberos configuration file. | +| `KerberosKeytabPath` | Kerberos keytab file. | +| `KerberosCredentialCachePath` | Kerberos credential cache. | +| `extraCredentials` | Extra credentials for connecting to external services. The extraCredentials is a list of key-value pairs. Example: `foo:bar;abc:xyz` will create credentials `abc=xyz` and `foo=bar` | +| `customHeaders` | Custom headers to inject through JDBC driver. The customHeaders is a list of key-value pairs. Example: `testHeaderKey:testHeaderValue` will inject the header `testHeaderKey` with value `testHeaderValue`. Values should be percent encoded. | diff --git a/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-redshift.md b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-redshift.md new file mode 100644 index 00000000..5350fd25 --- /dev/null +++ b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-redshift.md @@ -0,0 +1,65 @@ +# Connectivity to Redshift + +### Example URL + +``` +jdbc:redshift://redshift-cluster-name.kdkcis9g8.us-east-1.redshift.amazonaws.com:5439/dev +``` + +### Driver Name + +``` +com.amazon.redshift.jdbc.Driver +``` + +Amazon Redshift offers drivers for tools that are compatible with the JDBC 4.2 API. For information about the functionality supported by these drivers, see the [Amazon Redshift JDBC driver release notes](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.55.1083/Amazon+Redshift+JDBC+Release+Notes.pdf). + +For detailed information about how to install the JDBC driver version 1.0, reference the JDBC driver libraries, and register the driver class, see [Amazon Redshift JDBC driver installation and configuration guide](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.55.1083/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf). + +For each computer where you use the Amazon Redshift JDBC driver, make sure that Java Runtime Environment (JRE) 8.0 is installed. + +If you use the Amazon Redshift JDBC driver for database authentication, make sure that you have AWS SDK for Java 1.11.118 or later in your Java class path. If you don't have AWS SDK for Java installed, download the ZIP file with JDBC 4.2–compatible driver (without the AWS SDK) and driver dependent libraries for the AWS SDK: + +* [JDBC 4.2–compatible driver (without the AWS SDK) and driver dependent libraries for AWS SDK files version 1.2.55](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.55.1083/RedshiftJDBC42-1.2.55.1083.zip). + + The class name for this driver is `com.amazon.redshift.jdbc42.Driver`. + + This ZIP file contains the JDBC4.2–compatible driver (without the AWS SDK) and its dependent library files. Unzip the dependent jar files to the same location as the JDBC driver. Only the JDBC driver needs to be in the CLASSPATH because the driver manifest file contains all dependent library file names which are located in the same directory as the JDBC driver. For more information about how to install the JDBC driver, see [Amazon Redshift JDBC driver installation and configuration guide](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.55.1083/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf). + + Use this Amazon Redshift JDBC driver with the AWS SDK that is required for IAM database authentication. +* [JDBC 4.2–compatible driver (without the AWS SDK) version 1.2.55](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.55.1083/RedshiftJDBC42-no-awssdk-1.2.55.1083.jar). + + The class name for this driver is `com.amazon.redshift.jdbc42.Driver`. + + Be sure to use ANTLR version 4.8.1. The antlr4-runtime-4.8-1.jar is included in the ZIP download link above with the JDBC 4.2–compatible driver (without the AWS SDK) and driver dependent libraries for the AWS SDK. + +For more information about previous driver versions, see [Use previous JDBC driver versions with the AWS SDK for Java](https://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html#jdbc-previous-versions-with-sdk). + +Then download and review the [Amazon Redshift ODBC and JDBC driver license agreement](https://s3.amazonaws.com/redshift-downloads/drivers/Amazon+Redshift+ODBC+and+JDBC+Driver+License+Agreement.pdf). + +If your tool requires a specific previous version of a driver, see [Use previous JDBC driver version 1.0 driver versions in certain cases](https://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html#jdbc-previous-versions). + +### Getting the JDBC URL + +Before you can connect to your Amazon Redshift cluster from a SQL client tool, you need to know the JDBC URL of your cluster. The JDBC URL has the following format: `jdbc:redshift://endpoint`:`port`/`database`.Note + +A JDBC URL specified with the former format of `jdbc:postgresql://endpoint`:`port`/`database` still works. + +The fields of the format shown preceding have the following values. + +| Field | Value | +| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| `jdbc` | The protocol for the connection. | +| `redshift` | The subprotocol that specifies to use the Amazon Redshift driver to connect to the database. | +| `endpoint` | The endpoint of the Amazon Redshift cluster. | +| `port` | The port number that you specified when you launched the cluster. If you have a firewall, make sure that this port is open for you to use. | +| `database` | The database that you created for your cluster. | + +The following is an example JDBC URL: `jdbc:redshift://examplecluster.abc123xyz789.us-west-2.redshift.amazonaws.com:5439/dev` + +For information about how to get your JDBC connection, see [Finding your cluster connection string](https://docs.aws.amazon.com/redshift/latest/mgmt/configuring-connections.html#connecting-connection-string). + +If the client computer fails to connect to the database, you can troubleshoot possible issues. For more information, see [Troubleshooting connection issues in Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/mgmt/troubleshooting-connections.html).\ + + +[https://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html](https://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html) diff --git a/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-snowflake.md b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-snowflake.md new file mode 100644 index 00000000..0c6b9f80 --- /dev/null +++ b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-snowflake.md @@ -0,0 +1,77 @@ +# Connectivity to Snowflake + +### Example URL + +### JDBC Driver Connection String + +``` +jdbc:snowflake://accountname.us-east-2.aws.snowflakecomputing.com?db=cdq&warehouse=cdqw +``` + +### Driver Name + +``` +net.snowflake.client.jdbc.SnowflakeDriver +``` + +The previous driver class, `com.snowflake.client.jdbc.SnowflakeDriver`, is still supported but is deprecated (i.e. it will be removed in a future release, TBD). + +#### Connection Parameters + +For documentation on individual connection parameters, see the [JDBC Driver Connection Parameter Reference](https://docs.snowflake.com/en/user-guide/jdbc-parameters.html).`` + +Specifies the account identifier for your Snowflake account. For details, see [Account Identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html). For examples of the account identifier used in a JDBC connection string, see [Examples](https://docs.snowflake.com/en/user-guide/jdbc-configure.html#label-other-jdbc-connection-string-examples).`` + +Specifies a series of [one or more parameters](https://docs.snowflake.com/en/user-guide/jdbc-parameters.html), in the form of `=`, with each parameter separated by the ampersand character (`&`), and no spaces anywhere in the connection string. + +For documentation on individual connection parameters, see the [JDBC Driver Connection Parameter Reference](https://docs.snowflake.com/en/user-guide/jdbc-parameters.html). + +#### Other Parameters + +Any session parameter can be included in the connection string. For example: + +> `CLIENT_SESSION_KEEP_ALIVE=` +> +> Specifies whether to keep the current session active after a period of inactivity, or to force the user to login again. If the value is `true`, Snowflake keeps the session active indefinitely, even if there is no activity from the user. If the value is `false`, the user must log in again after four hours of inactivity. +> +> Default is `false`. + +For descriptions of all the session parameters, see [Parameters](https://docs.snowflake.com/en/sql-reference/parameters.html). + +#### Examples + +The following is an example of the connection string that uses an [account identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html) that specifies the account `myaccount` in the organization `myorganization`. + +> ``` +> jdbc:snowflake://myorganization-myaccount.snowflakecomputing.com/?user=peter&warehouse=mywh&db=mydb&schema=public +> ``` + +The following is an example of a connection string that uses the [account locator](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#label-account-locator) `xy12345` as the account identifier: + +> ``` +> jdbc:snowflake://xy12345.snowflakecomputing.com/?user=peter&warehouse=mywh&db=mydb&schema=public +> ``` + +Note that this example uses an account in the AWS US West (Oregon) region. If the account is in a different region or if the account uses a different cloud provider, you need to [specify additional segments after the account locator](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#label-account-locator). + +{% embed url="https://docs.snowflake.com/en/user-guide/jdbc-configure.html" %} + +### Private Link + +Please let us know if you are using private link for Snowflake. Setup can vary depending on the endpoint that is created. In most cases, use the private endpoint as a normal JDBC connection. + +{% embed url="https://community.snowflake.com/s/article/AWS-PrivateLink-and-Snowflake-detailed-troubleshooting-Guide" %} + +#### Advanced Private Link and Proxy + +Here is an example JDBC string connection we used that take into account the following setup: + +* \ is the full link to the Snowflake instance with the private link. +* DQ installed on-prem in a private IaaS and DQ is behind a proxy. +* If the Snowflake instance is using a private link, whitelist the private link URL to bypass the proxy. +* In addition to connectivity to the Snowflake instance, the JDBC driver tries to access Snowflake Blob storage by connecting directly to some S3 buckets managed by Snowflake. +* Those need to be whitelisted as well. + +#### Example URL + +jdbc:snowflake://\/?tracing=all\&useProxy=true\&proxyHost=10.142.22.37\&proxyPort=8080\&proxyUser=xyz\&proxyPassword=xyz\&nonProxyHosts=\*.[privatelink.snowflakecomputing.com](http://privatelink.snowflakecomputing.com)%[7Csfc-eu-ds1-customer-stage.s3.eu-central-1.amazonaws.com](http://7csfc-eu-ds1-customer-stage.s3.eu-central-1.amazonaws.com) diff --git a/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-sql-server.md b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-sql-server.md new file mode 100644 index 00000000..c09f172b --- /dev/null +++ b/connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-sql-server.md @@ -0,0 +1,49 @@ +# Connectivity to SQL Server + +### Example URL + +``` + jdbc:sqlserver://$host:1433;databaseName=ContosoRetailDW +``` + +### Driver Name + +``` +com.microsoft.sqlserver.jdbc.SQLServerDriver +``` + +### Setup + +1. Navigate to the connections page +2. Locate the SQL Server template +3. Fill in the required JDBC details +4. Click save to validate the connection + +![](<../../.gitbook/assets/image (73).png>) + +{% embed url="https://docs.microsoft.com/en-us/sql/connect/jdbc/microsoft-jdbc-driver-for-sql-server" %} + +### Troubleshooting + +The Microsoft JDBC Driver for SQL Server requires that TCP/IP be installed and running to communicate with your SQL Server database. You can use the SQL Server Configuration Manager to verify which network library protocols are installed. + +A database connection attempt might fail for many reasons. These can include the following: + +* TCP/IP is not enabled for SQL Server, or the server or port number specified is incorrect. Verify that SQL Server is listening with TCP/IP on the specified server and port. This might be reported with an exception similar to: "The login has failed. The TCP/IP connection to the host has failed." This indicates one of the following: + * SQL Server is installed but TCP/IP has not been installed as a network protocol for SQL Server by using the SQL Server Network Utility for SQL Server 2000 (8.x), or the SQL Server Configuration Manager for SQL Server 2005 (9.x) and later. + * TCP/IP is installed as a SQL Server protocol, but it is not listening on the port specified in the JDBC connection URL. The default port is 1433, but SQL Server can be configured at product installation to listen on any port. Make sure that SQL Server is listening on port 1433. Or, if the port has been changed, make sure that the port specified in the JDBC connection URL matches the changed port. For more information about JDBC connection URLs, see [Building the connection URL](https://docs.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url?view=sql-server-ver15). + * The address of the computer that is specified in the JDBC connection URL does not refer to a server where SQL Server is installed and started. + * The networking operation of TCP/IP between the client and server running SQL Server is not operable. You can check TCP/IP connectivity to SQL Server by using telnet. For example, at the command prompt, type `telnet 192.168.0.0 1433` where 192.168.0.0 is the address of the computer that is running SQL Server and 1433 is the port it is listening on. If you receive a message that states "Telnet cannot connect," TCP/IP is not listening on that port for SQL Server connections. Use the SQL Server Network Utility for SQL Server 2000 (8.x), or the SQL Server Configuration Manager for SQL Server 2005 (9.x) and later to make sure that SQL Server is configured to use TCP/IP on port 1433. + * The port that is used by the server has not been opened in the firewall. This includes the port that is used by the server or optionally, the port associated with a named instance of the server. +* The specified database name is incorrect. Make sure that you are logging on to an existing SQL Server database. +* The user name or password is incorrect. Make sure that you have the correct values. +* When you use SQL Server Authentication, the JDBC driver requires that SQL Server is installed with SQL Server Authentication, which is not the default. Make sure that this option is included when you install or configure your instance of SQL Server. + +### See also + +[Diagnosing problems with the JDBC driver](https://docs.microsoft.com/en-us/sql/connect/jdbc/diagnosing-problems-with-the-jdbc-driver?view=sql-server-ver15)\ +[Connecting to SQL Server with the JDBC driver](https://docs.microsoft.com/en-us/sql/connect/jdbc/connecting-to-sql-server-with-the-jdbc-driver?view=sql-server-ver15) + +### FAQ + +SQL Server represents RDS, Azure SQL, and traditional SQL Server installations diff --git a/dq-job-examples/data-quality-pipelines/README.md b/dq-job-examples/data-quality-pipelines/README.md new file mode 100644 index 00000000..74e85561 --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/README.md @@ -0,0 +1,5 @@ +# Pipelines + +## Programmatic DQ + +Don't like leaving your notebook? Want to build DQ into your in-house data quality pipeline? Collibra DQ can do both! diff --git a/dq-job-examples/data-quality-pipelines/aws-databricks-dq-pipeline.md b/dq-job-examples/data-quality-pipelines/aws-databricks-dq-pipeline.md new file mode 100644 index 00000000..580adb0b --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/aws-databricks-dq-pipeline.md @@ -0,0 +1,148 @@ +# AWS DataBricks + +## Getting started + +First use vanilla spark code to setup connection properties and access a database table via spark jdbc. Entire code example available at the end for copy paste. + +![](../../.gitbook/assets/dv-jdbc2.png) + +#### Schema output, Row Count and Runtime + +```scala +root + |-- EXCH: string (nullable = true) + |-- SYMBOL: string (nullable = true) + |-- TRADE_DATE: date (nullable = true) + |-- OPEN: decimal(9,3) (nullable = true) + |-- HIGH: decimal(9,3) (nullable = true) + |-- LOW: decimal(9,3) (nullable = true) + |-- CLOSE: decimal(9,3) (nullable = true) + |-- VOLUME: integer (nullable = true) + |-- PART_DATE_STR: date (nullable = true) + +Row Count: 102,817 +Runtime: 00:00:03 +``` + +## Next Configure Owl Options and Point to Owl Metastore + +This requires that you have imported the Owl libraries into your notebook or databricks env. + +![](<../../.gitbook/assets/Screen Shot 2020-05-09 at 11.02.15 PM.png>) + +## Next Run a Profile + +![](<../../.gitbook/assets/Screen Shot 2020-05-09 at 11.04.23 PM.png>) + +```scala ++-------------+-----+-------+-----------+--------+-----------+------+----+------+-------+-------+------+------+---------+ +| column|nulls|empties|cardinality|is_mixed|mixed_ratio| Int|Long|String|Decimal|Boolean|Double| Date|Timestamp| ++-------------+-----+-------+-----------+--------+-----------+------+----+------+-------+-------+------+------+---------+ +| HIGH| 0| 0| 19159| false| 0.0| 0| 0| 0| 102817| 0| 0| 0| 0| +| SYMBOL| 0| 0| 3137| false| 0.0| 0| 0|102817| 0| 0| 0| 0| 0| +| LOW| 0| 0| 18845| false| 0.0| 0| 0| 0| 102817| 0| 0| 0| 0| +| VOLUME| 0| 0| 25856| false| 0.0|102817| 0| 0| 0| 0| 0| 0| 0| +| TRADE_DATE| 0| 0| 33| false| 0.0| 0| 0| 0| 0| 0| 0|102817| 0| +| EXCH| 0| 0| 2| false| 0.0| 0| 0|102817| 0| 0| 0| 0| 0| +| CLOSE| 0| 0| 15781| false| 0.0| 0| 0| 0| 102817| 0| 0| 0| 0| +|PART_DATE_STR| 0| 0| 33| false| 0.0| 0| 0| 0| 0| 0| 0|102817| 0| +| OPEN| 0| 0| 16013| false| 0.0| 0| 0| 0| 102817| 0| 0| 0| 0| ++-------------+-----+-------+-----------+--------+-----------+------+----+------+-------+-------+------+------+---------+ +``` + +## Next Check for Duplicates + +Notice there is a duplicate discovered. NYSE AAN record exists twice in the 10/1/2018. This should not happen as end of day stock data should only have 1 record per stock symbol. Great DQ finding. + +![](<../../.gitbook/assets/Screen Shot 2020-05-09 at 11.06.25 PM.png>) + +## Next Scan for Outliers + +Notice that KOD.w the camera company Kodak commonly trades at less than 2 pennies and jumps to $2.35. Absolutely an outlier. This was a news event named Kodak coin, google it. + +![](<../../.gitbook/assets/Screen Shot 2020-05-09 at 11.07.54 PM.png>) + +```scala ++-----+------+---------+----------+----------+ +| key|column| value|prediction|confidence| ++-----+------+---------+----------+----------+ +|TPG.E|VOLUME| 23400.0| 0.0| 0| +|MTB-C|VOLUME| 0.0| 100.0| 0| +|KOD.W| OPEN| 2.35| 0.015| 1| +``` + +## Entire Code Snippet + +```scala +//--- GCP Postgres Connection ---// +val url = "jdbc:postgresql://${host}:5432/postgres?currentSchema=owl_test" +var connectionProps = new java.util.Properties() +connectionProps.setProperty("driver", "org.postgresql.Driver") +connectionProps.setProperty("user", "${user}") +connectionProps.setProperty("password", "${pass}") +connectionProps.setProperty("connectionUrl", url) + +//--- Load DataFrame From GCP Postgres ---// +val jdbcDF2 = spark.read.jdbc(url, "owl_test.nyse", connectionProps) +jdbcDF2.printSchema +jdbcDF2.cache +jdbcDF2.count + +//--- Owl Library Imports ---// +import com.owl.common.options._ +import com.owl.core.Owl +import com.owl.core.util.OwlUtils + +val opt = new OwlOptions() +//--- Owl Metastore ---// +opt.host = s"${host}" +opt.port = s"5432/postgres?currentSchema=public" +opt.pgUser = s"$user" +opt.pgPassword = s"$pass" + +//--- Run Options ---// +opt.dataset = "nyse_notebook_pipeline" +opt.runId = "2018-01-10" +opt.datasetSafeOff = true + +opt.dupe.on = true +opt.dupe.lowerBound = 99 +opt.dupe.include = Array("SYMBOL", "EXCH") + +opt.outlier.on = true +opt.outlier.lookback = 6 +opt.outlier.dateColumn = "TRADE_DATE" +opt.outlier.timeBin = OutlierOpt.TimeBin.DAY +opt.outlier.key = Array("SYMBOL") +opt.outlier.measurementUnit = "VOLUME=100000000,HIGH=0.1,LOW=0.1,OPEN=0.1,CLOSE=0.1" + +//--- Initialize Owl ---// +val currentDay = jdbcDF2.where(s"TRADE_DATE = '${opt.runId}' ") +val owl = OwlUtils.OwlContextWithHistory(dfCurrent = currentDay, + dfHist = jdbcDF2, opt = opt) + +//--- Pipeline Cmds ---// +owl.register(opt) +val profile = owl.profileDF +val outliers = owl.outliersDF +val dupes = owl.dupesDF +``` + +### Required configuration + +* **DataBricks Runtime**: 5.4 (includes Apache Spark 2.4.3, Scala 2.11) +* **Python Version**: 3 + +### Required libraries + +* **OWL jar file**\ + _Example: owl\_core\_trunk\_jar\_with\_dependencies.jar_\ + __ +* **JDBC driver:** org.springframework:spring-jdbc:4.3.16.RELEASE\ + +* **Database specific JDBC connector drivers**\ + _Example#1: mysql:mysql-connector-java:8.0.17_\ + _Example#2: org.postgresql:postgresql:jar:42.2.8_ + + + diff --git a/dq-job-examples/data-quality-pipelines/azure-databricks-dq-pipeline.md b/dq-job-examples/data-quality-pipelines/azure-databricks-dq-pipeline.md new file mode 100644 index 00000000..162e73cf --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/azure-databricks-dq-pipeline.md @@ -0,0 +1,37 @@ +# Azure DataBricks + +### Run a DQ check on any file in Azure Blob + +Read the File by setting up the azure key. + +```scala +spark.conf.set("fs.azure.account.key.abcCompany.blob.core.windows.net","GBB6Upzj4AxQld7cFv7wBYNoJzIp/WEv/5NslqszY3nAAlsalBNQ==") + +val df = spark.read.parquet("wasbs://company-abc@abceCompany.blob.core.windows.net/FILE_NAME/20190201_FILE_NAME.parquet") + +``` + +Process the file using Owl + +```scala +// register in Owl Catalog, Optional +val owl = new Owl(df).register + +// run a full DQ Check +owl.owlCheck() +``` + +Additional imports and input options + +```scala +import com.owl.core._ +import com.owl.common._ + +val props = new Props() +props.dataset = datasetName +props.runId = 2019-03-02 +props..... // look at the many input options +``` + +{% embed url="https://owl-analytics.com/Create-a-Data-Quality-Pipeline-using-Owl.html" %} + diff --git a/dq-job-examples/data-quality-pipelines/frequently-asked-questions.md b/dq-job-examples/data-quality-pipelines/frequently-asked-questions.md new file mode 100644 index 00000000..1ca87c1b --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/frequently-asked-questions.md @@ -0,0 +1,32 @@ +# FAQs + +## How to specify OWL database connection properties + +{% tabs %} +{% tab title="OwlOptions" %} +```scala +import com.owl.common.options.OwlOptions + +val opt = new OwlOptions() +opt.dataset = "" +opt.runId = "" // YYYY-MM-DD +opt.load.pguser = "" +opt.load.pgpassword = "" +opt.load.pghost = ":/" +``` +{% endtab %} + +{% tab title="Props" %} +```scala +import com.owl.common.Props + +val props = new Props() +props.dataset = "" +props.runId = "" // YYYY-MM-DD +props.pguser = "" +props.pgpassword = "" +props.host = ":/" +``` +{% endtab %} +{% endtabs %} + diff --git a/dq-job-examples/data-quality-pipelines/notebook-colmatch-example.md b/dq-job-examples/data-quality-pipelines/notebook-colmatch-example.md new file mode 100644 index 00000000..8de87826 --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/notebook-colmatch-example.md @@ -0,0 +1,120 @@ +# Column Match + +This example shows how one can get column level match statistics across datasources in an Owl Notebook. Supports exact and fuzzy matching. + +## Set ColMatch Parameters + +```scala + %spark + import com.owl.common.domain._ + import com.owl.common.Props + import com.owl.core.util.OwlUtils + import scala.collection.JavaConverters._ + import com.owl.common.Utils + val c1 = new Connection() + c1.dataset = "silo.account" + c1.user = "user" + c1.password = "pass" + c1.query = "select id, networth, acc_name, acc_branch from silo.account limit 200000" + c1.url = "jdbc:mysql://:3306" + + val c2 = new Connection() + c2.dataset = "silo.user_account" + c2.user = "user" + c2.password = "pass" + c2.query = "SELECT acc_name, acc_branch, networth FROM silo.account limit 200000" + c2.url = "jdbc:mysql://:3306" + + val c3 = new Connection() + c3.dataset = "silo.user_account" + c3.user = "user" + c3.password = "pass" + c3.query = "SELECT acc_name as acc_name2, acc_branch, networth FROM silo.account limit 100000" + c3.url = "jdbc:mysql://:3306" + + props.dataset = "colMatchTest1" + props.runId = "2017-02-04" + props.connectionList = List(c1,c2,c3).asJava + props.colMatchBatchSize = 2 + props.colMatchDurationMins = 3 + val owl = OwlUtils.OwlContext(spark.emptyDataFrame, props) +``` + +### Exact Match + +```scala +%spark +props.colMatchLevel = "exact" +owl.register(props) +owl.colMatchDF().show +``` + +#### Sample Result + +```scala ++------------+-----------------+----------+----------+---------------+ +| dataset_1| dataset_2| col_1| col_2|matchPercentage| ++------------+-----------------+----------+----------+---------------+ +|silo.account|silo.user_account| id| acc_name| 0| +|silo.account|silo.user_account| id|acc_branch| 0| +|silo.account|silo.user_account| id| networth| 0| +|silo.account|silo.user_account| id| owl_id| 0| +|silo.account|silo.user_account| networth| acc_name| 0| +|silo.account|silo.user_account| networth|acc_branch| 16| +|silo.account|silo.user_account| networth| networth| 100| +|silo.account|silo.user_account| networth| owl_id| 0| +|silo.account|silo.user_account| acc_name| acc_name| 87| +|silo.account|silo.user_account| acc_name|acc_branch| 0| +|silo.account|silo.user_account| acc_name| networth| 0| +|silo.account|silo.user_account| acc_name| owl_id| 0| +|silo.account|silo.user_account|acc_branch| acc_name| 0| +|silo.account|silo.user_account|acc_branch|acc_branch| 87| +|silo.account|silo.user_account|acc_branch| networth| 12| +|silo.account|silo.user_account|acc_branch| owl_id| 0| +|silo.account|silo.user_account| owl_id| acc_name| 0| +|silo.account|silo.user_account| owl_id|acc_branch| 0| +|silo.account|silo.user_account| owl_id| networth| 0| +|silo.account|silo.user_account| owl_id| owl_id| 0| ++------------+-----------------+----------+----------+---------------+ +only showing top 20 rows +``` + +### Fuzzy Match + +```scala +%spark +props.colMatchLevel = "fuzzy" +props.colMatchFuzzyDistance = 4 +owl.register(props) +owl.colMatchDF().show +``` + +#### Sample Result + +```scala ++------------+-----------------+----------+----------+---------------+ +| dataset_1| dataset_2| col_1| col_2|matchPercentage| ++------------+-----------------+----------+----------+---------------+ +|silo.account|silo.user_account| id| acc_name| 5| +|silo.account|silo.user_account| id|acc_branch| 27| +|silo.account|silo.user_account| id| networth| 22| +|silo.account|silo.user_account| id| owl_id| 0| +|silo.account|silo.user_account| networth| acc_name| 100| +|silo.account|silo.user_account| networth|acc_branch| 233| +|silo.account|silo.user_account| networth| networth| 200| +|silo.account|silo.user_account| networth| owl_id| 0| +|silo.account|silo.user_account| acc_name| acc_name| 162| +|silo.account|silo.user_account| acc_name|acc_branch| 262| +|silo.account|silo.user_account| acc_name| networth| 75| +|silo.account|silo.user_account| acc_name| owl_id| 0| +|silo.account|silo.user_account|acc_branch| acc_name| 262| +|silo.account|silo.user_account|acc_branch|acc_branch| 612| +|silo.account|silo.user_account|acc_branch| networth| 175| +|silo.account|silo.user_account|acc_branch| owl_id| 0| +|silo.account|silo.user_account| owl_id| acc_name| 0| +|silo.account|silo.user_account| owl_id|acc_branch| 0| +|silo.account|silo.user_account| owl_id| networth| 0| +|silo.account|silo.user_account| owl_id| owl_id| 0| ++------------+-----------------+----------+----------+---------------+ +only showing top 20 rows +``` diff --git a/dq-job-examples/data-quality-pipelines/notebook-outlier-example.md b/dq-job-examples/data-quality-pipelines/notebook-outlier-example.md new file mode 100644 index 00000000..94a60231 --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/notebook-outlier-example.md @@ -0,0 +1,139 @@ +# Outliers + +This real life use-case is when you have a large file or data frame with many days of data but you want the run profile to be the current day so that it trends properly overtime. Another nuance to this use-case is that the customer\_id is a unique field to the user and it should not show up in the analytics i.e. an outlier. But the customer\_id should be available when the user wants to query the rest api end points. The customer\_id is then used to link back the users original dataset. A bloomberg\_Id (BB\_ID) is a common example. + +### CSV File + +``` +fname,app_date,age,customer_id +Kirk,2018-02-24,18,31 +Kirk,2018-02-23,11,4 +Kirk,2018-02-22,10,3 +Kirk,2018-02-21,12,2 +Kirk,2018-02-20,10,1 +``` + +### Notebook Code (Spark Scala) + +```scala +val filePath = getClass.getResource("/notebooktest.csv").getPath + +val spark = SparkSession.builder + .master("local") + .appName("test") + .getOrCreate() + +val opt = new OwlOptions() +opt.dataset = "dataset_outlier" +opt.runId = "2018-02-24" +opt.outlier.on = true +opt.outlier.key = Array("fname") +opt.outlier.dateColumn = "app_date" +opt.outlier.timeBin = OutlierOpt.TimeBin.DAY +opt.outlier.lookback = 5 +opt.outlier.excludes = Array("customer_id") + +val dfHist = OwlUtils.load(filePath = filePath, delim = ",", sparkSession = spark) +val dfCurrent = dfHist.where(s"app_date = '${opt.runId}' ") + +val owl = OwlUtils.OwlContextWithHistory(dfCurrent=dfCurrent, dfHist=dfHist, opt=opt) +owl.register(opt) +owl.owlCheck() +``` + +### Owl Web UI + +Score drops from 100 to 99 based on the single outlier in the file. Row count is 1 because there is only 1 row in the current data frame. The historical data frame was provided for context and you can see those rows in the outlier drill-in. The customer\_id is available in the data preview and can be used as an API hook to link back to the original dataset. + +![](../../.gitbook/assets/owl-df-with-hist-customer\_id.png) + +After you run an owlcheck using owl.owlcheck you might want to check individual scores to see what type of issues were in the data. Owl can send back the records with issues in the format of a DataFrame using the notebook cmds or JSON from the REST api. + +```scala +val hoot = owl.hoot + +println(s"SHAPE ${hoot.shapeScore} ") +println(s"DUPE ${hoot.dupeScore} ") +println(s"OUTLIER ${hoot.outlierScore} ") +println(s"PATTERN ${hoot.patternScore} ") +println(s"RECORD ${hoot.recordScore} ") +println(s"SCHEMA ${hoot.schemaScore} ") +println(s"BEHAVIOR${hoot.behaviorScore} ") +println(s"SOURCE ${hoot.sourceScore} ") +println(s"RULES ${hoot.ruleScore} ") + +if (hoot.shapeScore > 0) { + owl.getShapeRecords.show +} +if (hoot.dupeScore > 0) { + owl.getDupeRecords.show +} +``` + +``` ++-------+---------+--------------------+--------+-----------+-------+------+ +|row_cnt|obs_score| row_key|obs_type|customer_id| fname|owl_id| ++-------+---------+--------------------+--------+-----------+-------+------+ +| 21| 46|afa89984ce472a409...| DUPE| 32| Kirk| 1| +| 22| 46|afa89984ce472a409...| DUPE| 31|Kirk's.| 2| +| 23| 60|41ea2d828b1a5fbf2...| DUPE| 30| Dan| 3| +| 24| 60|41ea2d828b1a5fbf2...| DUPE| 27| Dan| 6| +``` + +``` ++---------------+--------------------+--------+----------+--------------+--------+-------+-------+---+--------------------+-----------+-------+------+--------+ +| dataset| run_id|col_name|col_format|col_format_cnt|owl_rank|row_cnt|row_key|age| app_date|customer_id| fname|owl_id|time_bin| ++---------------+--------------------+--------+----------+--------------+--------+-------+-------+---+--------------------+-----------+-------+------+--------+ +|dataset_outlier|2018-02-24 00:00:...| fname| xxxx'x.| 1| 1| 2|xxxx'x.| 18|2018-02-24 00:00:...| 31|Kirk's.| 2| null| ++---------------+--------------------+--------+----------+--------------+--------+-------+-------+---+--------------------+-----------+-------+------+--------+ +``` + +{% swagger baseUrl="http://$host" path="/v2/getoutlier?dataset=dataset_outlier&runId=2018-02-24" method="get" summary="GetOutlier" %} +{% swagger-description %} + +{% endswagger-description %} + +{% swagger-parameter in="path" name="dataset" type="string" %} +name of dataset +{% endswagger-parameter %} + +{% swagger-parameter in="path" name="dataset" type="string" %} +yyyy-MM-dd format can include time and timezone +{% endswagger-parameter %} + +{% swagger-response status="200" description="" %} +```bash +{ + confidence: 77 + dataset: "dataset_outlier" + keyArr: null + lb: 0 + outColumn: "age" + outKey: "Kirk" + outMedian: "10.5" + outValue: "18.0" + runId: "2018-02-24T05:00:00.000+0000" + ub: 0 +} +``` +{% endswagger-response %} +{% endswagger %} + +{% swagger baseUrl="http://$host" path="/v2/getdatashapes?dataset=dataset_outlier&runId=2018-02-24" method="get" summary="GetShape" %} +{% swagger-description %} + +{% endswagger-description %} + +{% swagger-parameter in="path" name="dataset" type="string" %} +name of dataset +{% endswagger-parameter %} + +{% swagger-parameter in="path" name="runId" type="string" %} +yyyy-MM-dd format can include time and timezone +{% endswagger-parameter %} + +{% swagger-response status="200" description="" %} +``` +``` +{% endswagger-response %} +{% endswagger %} diff --git a/dq-job-examples/data-quality-pipelines/owl-options-api.md b/dq-job-examples/data-quality-pipelines/owl-options-api.md new file mode 100644 index 00000000..3f6043ab --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/owl-options-api.md @@ -0,0 +1,187 @@ +--- +description: OwlOptions is the new standard for configuring Owl within a notebook. +--- + +# Options API + +## Field mappings + +| Field name | CLI prop | Description | +| -------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **dataset** | _ds_ | dataset name, example: userTable or users or user\_file | +| **rundId** | _rd_ | run date, must be in format **yyyy-MM-dd** or for incremental use Hours or Minutes **yyyy-MM-dd HH:mm** | +| **rundIdEnd** | _rdEnd_ | end date for query ranges **t\_date >= ${rd} and t\_date < ${rdEnd}**, must be in format **yyyy-MM-dd** or for incremental use Hours or Minutes **yyyy-MM-dd HH:mm** | +| **passFail** | _passfaillimit_ | Limit for passing or failing runs | +| **jobId** | | | +| **onReadOnly** | _readonly_ | Do not persist results to the Owl meta-store - useful during testing. | + +## Load Options + +| Field name | CLI prop | Description | +| ------------------- | ---------- | ------------------------------------------------------- | +| **fullFile** | _fullfile_ | use entire file for lookbacks instead of just filequery | +| **fileQuery** | | | +| **header** | | | +| **headerSrc** | | | +| **hasHeader** | | | +| **isParallel** | | | +| **isJson** | | | +| **isMixedJson** | | | +| **isMapsJson** | | | +| **flatten** | | | +| **isMultiLine** | | | +| **sparkprinc** | | | +| **sparkkeytab** | | | +| **jdbcprinc** | | | +| **jdbckeytab** | | | +| **srcpwdmgr** | | | +| **pwdmgr** | | | +| **pguser** | pguser | | +| **pgpassword** | pgpassword | | +| **pghost** | host | | +| **pgport** | port | | +| **executorcores** | | | +| **isParquet** | | | +| **isAvro** | | | +| **avroSchema** | | | +| **isXml** | | | +| **xmlRowTag** | | | +| **isOrc** | | | +| **dateFmt** | | | +| **timeFmt** | | | +| **datasetSafety** | | | +| **filePath** | | | +| **delimiter** | | | +| **fileLookBack** | | | +| **dbLookBack** | | | +| **connectionURL** | | | +| **userName** | | | +| **password** | | | +| **sqlQuery** | | | +| **connectionProps** | | | +| ~~zkHost~~ | ~~~~ | _**Deprecated**_ | +| ~~zkPort~~ | ~~~~ | _**Deprecated**_ | +| ~~zkPath~~ | ~~~~ | _**Deprecated**_ | + +## Outlier Options + +| Field name | CLI prop | Description | +| ---------------- | -------- | ------------------------------------------------------------- | +| **on** | dl | deep learning | +| **lookback** | dllb | deep learning lookback example value 5, for 5 days | +| **key** | dlkey | deep learning key. comma delim key ex: symbol,date | +| **dateField** | | | +| **bin** | | | +| **includes** | dlinc | deep learning col limit, example : open,close,high,volume | +| **excludes** | dlexc | deep learning col exclusion, example : open,close,high,volume | +| **categorical** | | | +| **by** | | | +| **limit** | | | +| **historyLimit** | | | +| **score** | | | + +## FPG Options + +| Field name | CLI prop | Description | +| --------------- | ------------- | --------------------------------------------------------------------- | +| **on** | fpgon | pattern mining | +| **lookback** | fpglb | lookback interval for pattern mining. Ex: **-fpglb 5** | +| **key** | fpgkey | natural key for pattern mining activity | +| **dateField** | fpgdc | date column for pattern mining. Ex: **-fpgdc date\_col** | +| ~~**lowFreq**~~ | | _**Deprecated**_ | +| **includes** | fpginc | pattern mining is expensive use this input to limit the observed cols | +| **excludes** | fpgexc | pattern mining is expensive use this input to limit the observed cols | +| **timeBin** | fpgtbin | time bin for pattern mining. Ex: **-fpgtbin DAY** | +| **score** | fpgscore | score for pattern mining records | +| **minSupport** | fpgsupport | | +| **confidence** | fpgconfidence | | + +## Dupe Options + +| Field name | CLI prop | Description | +| --------------- | ---------- | ----------------------------------------------------- | +| **on** | dupe | duplicate record detection | +| **includes** | dupeinc | duplicate record detection, column inclusion list | +| **excludes** | dupeexc | duplicate record detection, column exclusion list | +| **depth** | | | +| **lowerBound** | dupelb | duplicate lower bounds on percent match default \[85] | +| **upperBound** | | | +| **blocksize** | | | +| **useCache** | | | +| **checkHeader** | | | +| **exactMatch** | | | +| **ignoreCase** | dupenocase | duplicate record detection, column exclusion list | +| **score** | dupescore | | +| **limit** | dupelimit | Limit for dupe rows stored | + +## Profile Options + +| Field name | CLI prop | Description | +| -------------------- | -------- | ----------- | +| **on** | | | +| **includes** | | | +| **excludes** | | | +| **dataShapeOn** | | | +| **statsOn** | | | +| **correlationOn** | | | +| **histogramOn** | | | +| **cardinalityOn** | | | +| **dataShapeColsInc** | | | +| **dataShapeColsExc** | | | + +## Source Options + +| Field name | CLI prop | Description | +| ---------------------- | -------- | ----------- | +| **on** | | | +| **includes** | | | +| **excludes** | | | +| **key** | | | +| **fileQuery** | | | +| **map** | | | +| **score** | | | +| **datasetSrc** | | | +| **driverSrc** | | | +| **userNameSrc** | | | +| **passwordSrc** | | | +| **connectionURLSrc** | | | +| **sqlQuerySrc** | | | +| **connectionPropsSrc** | | | + +## Rule Options + +| Field name | CLI prop | Description | +| ------------- | -------- | ----------- | +| **on** | | | +| **rulesOnly** | | | +| **semantic** | | | + +## ColMatch Options + +| Field name | CLI prop | Description | +| ----------------------------- | -------- | ----------- | +| **colMatchParallelProcesses** | | | +| **colMatchDurationMins** | | | +| **colMatchBatchSize** | | | +| **connectionList** | | | + +## Spark Options + +| Field name | CLI prop | Description | +| ------------------ | -------- | ----------- | +| **numExecutors** | | | +| **executorMemory** | | | +| **driverMemory** | | | +| **executorCores** | | | +| **master** | | | +| **jars** | | | +| **libs** | | | +| **driver** | | | + +## Misc Options + +| Field name | CLI prop | Description | +| ------------- | -------- | ----------- | +| **obslimit** | | | +| **nullValue** | | | + diff --git a/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/README.md b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/README.md new file mode 100644 index 00000000..5dd43c53 --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/README.md @@ -0,0 +1,73 @@ +--- +description: >- + In this section you can learn how to work with Rules in Notebooks written in + Scala. +--- + +# Rules + +## Instantiation + +| Code | Description | +| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | +| val rule = new Rule() | Instantiating a Rule object | +| rule.setDataset(_\_) | Adding the name of the dataset | +| rule.setRuleNm(_\_) | Adding the name of the given rule | +| rule.setRuleValue(_\_) | Setting the simple _RULE\_EXPRESSION_ | +| rule.setRuleType(_\_) | Setting the rule type | +| rule.setPerc(_\_) | Setting the percentage | +| rule.setPoints(_\_) |

Setting how many points
will be deducted from total
for each percentage

| +| rule.setIsActive(_\_) |

Making rule active/inactive

Possible values:

  • ACTIVE: 1 / true
  • INACTIVE: 0 / false
| +| rule.setUserNm(_\_) | Adding the owner | + +## Rule types + +{% content-ref url="sql-based-rules/" %} +[sql-based-rules](sql-based-rules/) +{% endcontent-ref %} + +{% content-ref url="global-rules.md" %} +[global-rules.md](global-rules.md) +{% endcontent-ref %} + +{% content-ref url="data-type-based-rules.md" %} +[data-type-based-rules.md](data-type-based-rules.md) +{% endcontent-ref %} + +## Requirements + +#### Required imports + +```scala +import com.owl.core.Owl +import com.owl.core.util.OwlUtils +import com.owl.common.options.OwlOptions +import com.owl.common.{Props, Utils} + +import com.owl.common.domain2.Rule + +import org.apache.spark.sql.functions._ +``` + +#### SparkSession initialization + +DataBricks and other notebook execution frameworks are working with managed/shared spark session, therefor we recommend to use this code snippet in your notebook to initialize the current spark session properly. + +```scala +//----- Init Spark ----- // +def sparkInit(): Unit = { + sparkSession = SparkSession.builder + .master("local") + .appName("test") + .getOrCreate() +} +``` + +{% hint style="danger" %} +Don't call **spark.stop** at any of your notebooks, otherwise the execution engine will exit immediately! +{% endhint %} + +{% hint style="warning" %} +Make sure OwlContext is **already** created before using any method from OwlUtils! +{% endhint %} + diff --git a/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/data-type-based-rules.md b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/data-type-based-rules.md new file mode 100644 index 00000000..3bd7fc69 --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/data-type-based-rules.md @@ -0,0 +1,61 @@ +--- +description: Simple check for individual columns +--- + +# Data type based rules + +## Rule types + +### Empty check + +**Rule type: EMPTYCHECK**\ +**Description:** Checking whether the target column has empty values or not + +### Null check + +**Rule type: NULLCHECK**\ +**Description:** Checking whether the target column has _NULL_ values or not + +### Date check + +**Rule type: DATECHECK**\ +**Description:** Checking whether the target column has **only** DATE values or not + +### Integer check + +**Rule type: INTCHECK**\ +**Description:** Checking whether the target column has **only** INTEGER values or not + +### Double check + +**Rule type: DOUBLECHECK**\ +**Description:** Checking whether the target column has **only** DOUBLE values or not + +### String check + +**Rule type: STRINGCHECK**\ +**Description:** Checking whether the target column has **only** STRING values or not. + +### Mixed datatype check + +**Rule type: DATATYPECHECK**\ +**Description:** --- + +## Syntax + +* **\** - Fixed key to the rule type +* **\** - Column to apply the rule +* **\** - Custom name of the rule + +```scala +opt.dataset = "example_ds" + +val rule = RuleBll.createRule(opt.dataset) +rule.setRuleNm("") +rule.setRuleValue("") +rule.setRuleType("") +rule.setPerc(1.0) +rule.setPoints(1) +rule.setIsActive(1) +rule.setUserNm("admin") +``` diff --git a/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/global-rules.md b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/global-rules.md new file mode 100644 index 00000000..73e68213 --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/global-rules.md @@ -0,0 +1,33 @@ +--- +description: >- + Distributed with the application by default and you can use them from + UI/CLI/Notebooks +--- + +# Global rules + +## Rule types + +* Invalid\_Email\_Check +* Invalid\_Phone\_Num\_Check +* Invalid\_Zip\_Code\_Check +* Invalid\_SSN\_Check +* Invalid\_IP\_Address\_Check +* Invalid\_Gender\_Check +* Invalid\_EIN\_Check +* Invalid\_State\_Check +* Invalid\_Credit\_Card\_Check +* Valid\_Email\_Check +* Valid\_Phone\_Num\_Check +* Valid\_Zip\_Code\_Check +* Valid\_SSN\_Check +* Valid\_IP\_Address\_Check +* Valid\_Gender\_Check +* Valid\_EIN\_Check +* Valid\_State\_Check +* Valid\_Credit\_Card\_Check +* Percent\_Move\_5 Percent\_Move\_10 +* Percent\_Move\_20 Percent\_Move\_50 +* Not\_In\_Previous\_run +* Not\_In\_Current\_run +* Having\_Count\_Greater\_Than\_One diff --git a/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/README.md b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/README.md new file mode 100644 index 00000000..af629aa6 --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/README.md @@ -0,0 +1,17 @@ +# SQL based rules + +## Simple rule + +#### Rule Type - SQLG + +{% content-ref url="simple-rule.md" %} +[simple-rule.md](simple-rule.md) +{% endcontent-ref %} + +## Freeform SQL + +#### RuleType - SQLF + +{% content-ref url="freeform-sql.md" %} +[freeform-sql.md](freeform-sql.md) +{% endcontent-ref %} diff --git a/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/freeform-sql.md b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/freeform-sql.md new file mode 100644 index 00000000..8ee44fb1 --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/freeform-sql.md @@ -0,0 +1,151 @@ +--- +description: >- + It would be used when applying a complex condition across multiple + tables/columns and generally when more flexibility/customization is desired. +--- + +# Freeform SQL + +## Individual statement + +#### Syntax + +```sql +SELECT * FROM @ +WHERE +GROUP BY +HAVING +``` + +The base of the statement is given with **@\** style. In general the \ is the same, where the rule is attached to, but basically you can use any valid dataset name in the expression. + +#### Examples + +{% tabs %} +{% tab title="Simple rule expression" %} +```scala +opt.dataset = "example_ds" + +val rule = RuleBll.createRule(opt.dataset) +rule.setRuleNm("is_city_not_null_or_empty") +rule.setRuleValue("select * from @example_ds t where t.amount > '5000'") +rule.setRuleType("SQLF") +rule.setPerc(1.0) +rule.setPoints(1) +rule.setIsActive(1) +rule.setUserNm("admin") +``` +{% endtab %} + +{% tab title="Complex rule expression" %} +```scala +opt.dataset = "unique_rule_ds" + +val rule = RuleBll.createRule(opt.dataset) +rule.setRuleNm("unique_rule") +rule.setRuleValue("select * from ( select count(*) as cnt, customer_id from @unique_rule_ds group by customer_id ) having cnt > 1") +rule.setRuleType("SQLF") +rule.setPerc(1.0) +rule.setPoints(1) +rule.setIsActive(1) +rule.setUserNm("admin") +``` +{% endtab %} + +{% tab title="RegExp expression" %} +```scala +opt.dataset = "regexp_rule_ds" + +val rule = RuleBll.createRule(opt.dataset) +rule.setRuleNm("LIKE_rule") +rule.setRuleValue("select * from @regexp_rule_ds.SYMBOL rlike '^ABB+' ") +rule.setRuleType("SQLG") +rule.setPerc(0.02) +rule.setPoints(1) +rule.setIsActive(1) +rule.setUserNm("admin") +``` +{% endtab %} +{% endtabs %} + +## Join statements + +#### A**vailable join types between multiple dataset** + +* WHERE tableA.id = tableB.id style +* INNER JOIN +* LEFT \ JOIN +* RIGHT \ JOIN + +#### **Joining other dataset** + +* Getting historical state of the same dataset\ + \ + **Syntax:** **@t\**,\ + where _n_ parameter means, how many days should we go back in the past at the base dataset (marked with @\)\ + \ + **Example:** + + * **@t1**, will point to the data which was used at yesterday's run + * **@t4**, will point to the data which was used 4 days ago + + +* Getting different dataset\ + **Syntax: @\**\ + **** + +### WHERE style + +{% tabs %} +{% tab title="Look-back dataset" %} +```sql +SELECT * FROM @ , @t1 [] +WHERE AND +``` + +#### Example + +```scala +opt.dataset = "example_ds" + +val rule = RuleBll.createRule(opt.dataset) +rule.setRuleValue("select * from @example_ds t, @t1 where t.customer_id = t1.customer_id and t.card_number <> t1.card_number ") +rule.setRuleType("SQLF") +``` +{% endtab %} + +{% tab title="Different dataset" %} +```sql +SELECT * FROM @ , @ [] +WHERE AND +``` + +#### Example + +```scala +opt.dataset = "example_ds" +opt2.dataset = "other_ds" + +val rule = RuleBll.createRule(opt.dataset) +rule.setRuleValue("select * from @example_ds t, @other_ds ds2 where t.customer_id = ds2.customer_id and t.card_number <> ds2.card_number ") +rule.setRuleType("SQLF") +``` +{% endtab %} +{% endtabs %} + +### LEFT JOIN + +**Example** + +```scala +opt.dataset = "example_ds" + +val rule = RuleBll.createRule(opt.dataset) +rule.setRuleNm("not_back2back_days") +rule.setRuleValue(" select * from @example_ds A LEFT OUTER JOIN @t1 B ON A.customer_id = B.customer_id where A.customer_id is not null and B.customer_id is null ") +rule.setRuleType("SQLF") +rule.setPerc(1.0) +rule.setPoints(1) +rule.setIsActive(1) +rule.setUserNm("admin") +``` diff --git a/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/simple-rule.md b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/simple-rule.md new file mode 100644 index 00000000..7b7a7021 --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/owl-rules-dq-pipeline/sql-based-rules/simple-rule.md @@ -0,0 +1,233 @@ +--- +description: >- + Simple rules would be applied to filter a condition on a single column in a + single table. +--- + +# Simple rule + +## Example #1 + +In this example you can see how to create a simple SQL rule, with name **simple\_sql\_rule**. + +| Code | Description | +| ------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| rule.setRuleNm("**simple\_sql\_rule**") | Adding the name of the given rule | +| rule.setRuleValue("**startDate < '2011-11-01'")** |

Setting the simple SQL expression.
No JOIN allowed between tables!

| +| rule.setRuleType("**SQLG**") | Setting the rule type | + +### Code + +{% code title="example_simple_sql_rule.scala" %} +```scala +import com.owl.core.Owl +import com.owl.core.util.OwlUtils +import com.owl.common.bll.{RuleBll, RuleTemplateBll} +import com.owl.common.domain2.Rule +import com.owl.common.options.{LoadOpt, OwlOptions} + +import org.junit.{Assert, Test} + +import org.apache.spark.sql.SparkSession +import org.apache.spark.sql.functions._ + +val loadOptions = new LoadOpt { + pghost = "localhost:5432/postgres" + pguser = "username" + pgpassword = "password" +} + +//----- Init Spark ----- // +def sparkInit(): SparkSession = { + val sparkSession = SparkSession.builder + .master("local") + .appName("test") + .getOrCreate() + + sparkSession +} + +@Test def simpleRule(): Unit = { + + // Arrange + val spark = sparkInit() + import spark.implicits._ + + val headers = "firstName,lastName,startDate" + val source = Seq( + ("Thomas", "Martinez", "2010-11-01"), + ("Harry", "Williams", "2012-05-01"), + ("Ethan", "Davis", "2009-08-01") + ) + val arr = headers.split(",") + val df = source.toDF(arr: _*) + + val opt = new OwlOptions { + runId = "2019-09-20" + dataset = "simple_sql_rule_ds" + onReadOnly = false + load = loadOptions + } + + val rule = new Rule { + setDataset(opt.dataset) + setRuleNm("simple_sql_rule") + setRuleValue("startDate < '2011-11-01'") + setRuleType("SQLG") + setPerc(1.0) + setPoints(1) + setIsActive(1) + setUserNm("admin") + } + + val owl = OwlUtils.OwlContext(df, opt) + .register(opt) + + OwlUtils.addRule(rule) + + // Act + owl.owlCheck() + + // Assert + import scala.collection.JavaConversions + val hootRule = JavaConversions.asScalaBuffer(owl.hoot.rules).find(x => rule.getRuleNm.equals(x.getRuleNm)).orNull + Assert.assertNotNull(hootRule) + Assert.assertEquals(66, hootRule.getScore) +} + +// Execute notebook +simpleRuleNotebook() +``` +{% endcode %} + +### Result + +#### via Code + +You can do multiple assertion on the result of the OwlCheck process.\ +Using **owl.hoot** parameter will provide access to the execution results, in this case for the rule + +#### via UI + +![Results of notebook execution on UI](<../../../../.gitbook/assets/image (11).png>) + +## Example #2 + +In this example you can see how to create a simple SQL with rule with **templates**, with name **simple\_sql\_rule\_with\_template**. + +#### Steps + +1. Create the rule template, where the template column name should be marked with **$colNm** string.\ + + + ```scala + val ruleTemplate = RuleTemplateBll.createRuleTemplate( + "not_null_or_empty", + "Column cannot contain null or empty values", + " $colNm is null or $colNm = \'\' or $colNm = \'null\' " + ) + ``` +2. Create the Rule instance, where value of **RuleValue** will be used to replace **$colNm** in the template expression.\ + + + ```scala + val rule = RuleBll.createRule(opt.dataset) + rule.setRuleNm("is_city_not_null_or_empty") + rule.setRuleValue("city") + rule.setRuleType("CUSTOM") // legacy type required to look into rule repo + rule.setRuleRepo("not_null_or_empty") // custom rule name to pull rule value from rule repo + rule.setPerc(1.0) + rule.setPoints(1) + rule.setIsActive(1) + rule.setUserNm("admin") + ``` + +### Code + +```scala +import com.owl.core.Owl +import com.owl.core.util.OwlUtils +import com.owl.common.bll.{RuleBll, RuleTemplateBll} +import com.owl.common.domain2.Rule +import com.owl.common.options.{LoadOpt, OwlOptions} + +import org.junit.{Assert, Test} + +import org.apache.spark.sql.SparkSession +import org.apache.spark.sql.functions._ + +val loadOptions = new LoadOpt { + pghost = "localhost:5432/postgres" + pguser = "username" + pgpassword = "password" +} + +//----- Init Spark ----- // +def sparkInit(): SparkSession = { + val sparkSession = SparkSession.builder + .master("local") + .appName("test") + .getOrCreate() + + sparkSession +} + +@Test def simpleRuleWithTemplate(): Unit = { + + // Arrange + val spark = sparkInit() + import spark.implicits._ + + val headers = "firstName,lastName,city" + val source = Seq( + ("Thomas", "Martinez", ""), + ("Harry", "Williams", null), + ("Ethan", "Davis", "Los Angeles") + ) + val arr = headers.split(",") + val df = source.toDF(arr: _*) + + val opt = new OwlOptions { + runId = "2019-09-20" + dataset = "simple_sql_rule_with_template_ds" + onReadOnly = false + load = loadOptions + } + + val ruleTemplate = RuleTemplateBll.createRuleTemplate("not_null_or_empty","Column cannot contain null or empty values", " $colNm is null or $colNm = \'\' or $colNm = \'null\' ") + + val rule = RuleBll.createRule(opt.dataset) + rule.setRuleNm("is_city_not_null_or_empty") + rule.setRuleValue("city") + rule.setRuleType("CUSTOM") // legacy type required to look into rule repo + rule.setRuleRepo("not_null_or_empty") // custom rule name to pull rule value from rule repo + rule.setPerc(1.0) + rule.setPoints(1) + rule.setIsActive(1) + rule.setUserNm("admin") + + val owl = OwlUtils.OwlContext(df, opt) + .register(opt) + + OwlUtils.addRuleTemplate(ruleTemplate) + OwlUtils.addRule(rule) + + // Act + owl.owlCheck() + + // Assert + import scala.collection.JavaConversions + val hootRule = JavaConversions.asScalaBuffer(owl.hoot.rules).find(x => rule.getRuleNm.equals(x.getRuleNm)).orNull + Assert.assertNotNull(hootRule) + Assert.assertEquals(66, hootRule.getScore) +} + +// Execute notebook +simpleRuleWithTemplate() +``` + +### Result + +#### via UI + +![](<../../../../.gitbook/assets/image (12).png>) diff --git a/dq-job-examples/data-quality-pipelines/simple-spark-data-pipeline.md b/dq-job-examples/data-quality-pipelines/simple-spark-data-pipeline.md new file mode 100644 index 00000000..14e2da09 --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/simple-spark-data-pipeline.md @@ -0,0 +1,112 @@ +# Simple + +## Load Table use SparkJDBC + +```scala +//--- Configure Table From Database ---// +val connProps = Map ( + "driver" -> "org.postgresql.Driver", + "user" -> s"${user}", + "password" -> s"${pass}", + "url" -> s"jdbc:postgresql://${host}:${port}/${database}", + "dbtable" -> "owl_test.nyse" +) + +//--- Load Spark DataFrame ---// +val jdbcDF = spark.read.format("jdbc").options(connProps).load +jdbcDF.show +``` + +## Configure Owl Options + +Connect to Owl's Metadata Database and control DQ scan options. Wrap sparkDF with Owl context. + +```scala +import com.owl.common.options._ +import com.owl.core.util.OwlUtils + +val opt = new OwlOptions +//--- Owl Metastore ---// +opt.host = s"$owlHost" +opt.port = s"5432/postgres?currentSchema=public" +opt.pgUser = s"$owlUser" +opt.pgPassword = s"$owlPass" +//--- Run Options ---// +opt.dataset = "owl_test.nyse" +opt.runId = "2018-01-10" +opt.datasetSafeOff = true + +val owl = OwlUtils.OwlContext(jdbcDF, opt) +``` + +## Register with Catalog and Run Profile + +```scala +//--- Register with Owl Catalog ---// +owl.register(opt) + +//--- Profile Dataset ---// +val profile = owl.profileDF +profile.show +``` + +Notice that Owl returns results as Dataframes. This is a fantastic abstraction that allows you to ignore all domain objects and custom types and interact with a scaleable generic result set using common protocols like "where" or "filter" or "save" or "write" all with parallel operations. + +```scala ++--------------+-----+-------+-----------+---+---+--------+-----------+------+----+------+-------+-------+------+----+---------+ +| column|nulls|empties|cardinality|min|max|is_mixed|mixed_ratio| Int|Long|String|Decimal|Boolean|Double|Date|Timestamp| ++--------------+-----+-------+-----------+---+---+--------+-----------+------+----+------+-------+-------+------+----+---------+ +| tenant_id| 0| 0| 60| 0| 9| false| 0.0|100000| 0| 0| 0| 0| 0| 0| 0| +| a11| 0| 0| 1|a11|a11| false| 0.0| 0| 0|100000| 0| 0| 0| 0| 0| +| a10| 0| 0| 1|a10|a10| false| 0.0| 0| 0|100000| 0| 0| 0| 0| 0| +| account_type| 0| 0| 3| 02| 06| false| 0.0|100000| 0| 0| 0| 0| 0| 0| 0| +| a13| 0| 0| 1|a13|a13| false| 0.0| 0| 0|100000| 0| 0| 0| 0| 0| +|security_alias| 0| 0| 3| 0| 2| false| 0.0|100000| 0| 0| 0| 0| 0| 0| 0| +| a12| 0| 0| 1|a12|a12| false| 0.0| 0| 0|100000| 0| 0| 0| 0| 0| ++--------------+-----+-------+-----------+---+---+--------+-----------+------+----+------+-------+-------+------+----+---------+ +``` + +### Profile UI + +While the spark DF.show() is a nice and convenient output format, you may prefer a rich UI visual that tracks the data tests over time. The UI provides trend analysis, data drift, data relationships and more. + +![](../../.gitbook/assets/owl-profiler.png) + +## Duplicates + +Take duplicate detection for example. A common use-case where a business wants to make sure they do not have repeated or duplicate records in a table. Set the lowerBound to the percent fuzzy match you are willing to accept, commonly 87% or higher is an interesting match. You might also want to target a single day or week or month that you shouldn't have dupes within. Notice the .where function and then pass in a custom dataframe to the Owl context. + +```scala +opt.dupe.on = true +opt.dupe.lowerBound = 99 +opt.dupe.include = Array("SYMBOL", "EXCH") + +val df1Day = jdbcDF.where("TRADE_DATE = '2018-01-10' ") +val owl = OwlUtils.OwlContext(df1Day, opt) + +val dupes = owl.dupesDF +dupes.show + +// rdd collect +dupes.rdd.collect.foreach(println) + +// records linked together for remediation +owl.getDupeRecords.show +``` + +## Outliers + +Gaining and understanding of your outliers is a commonly desired DQ function. Owl has several configurations to help find the most meaningful outliers in your dataset and over time. Below compares the current day to a baseline of days in the historical dataframe. + +```scala +opt.outlier.on = true +opt.outlier.lookback = 6 +opt.outlier.dateColumn = "TRADE_DATE" +opt.outlier.timeBin = OutlierOpt.TimeBin.DAY +opt.outlier.key = Array("SYMBOL") + +val df1Day = jdbcDF2.where("TRADE_DATE = '2018-01-10' ") +val owl = OwlUtils.OwlContextWithHistory(dfCurrent = df1Day, dfHist = jdbcDF2, opt = opt) +val outliers = owl.outliersDF +outliers.show +``` diff --git a/dq-job-examples/data-quality-pipelines/spark-dq-pipeline.md b/dq-job-examples/data-quality-pipelines/spark-dq-pipeline.md new file mode 100644 index 00000000..2864de92 --- /dev/null +++ b/dq-job-examples/data-quality-pipelines/spark-dq-pipeline.md @@ -0,0 +1,197 @@ +--- +description: Using Owl on Spark +--- + +# Advanced + +## Programmatic DQ + +Don't like leaving your notebook? Want to build Owl into your in-house data quality pipeline? Owl can do both! + +### Real World Examples + +### Rules + +Let's assume we were provided a file named "atm\_cust\_file" and want to load it into a database table as well as scan it for all possible errors. We want to provide a couple levels of protection. 1) A business rule checking if _a customer joined before before the company was founded_. 2) Check if the file 100% matches to the DataFrame or db table we've created. 3) Check for all possible outliers or anomalies in the dataset. Each one of these 3 issues had a different impact to the business and causes a different flow to trigger in our pipeline. + +#### Add Rule + +Let's create a simple rule and assign points to the overall scoring system for later delegation. + +```scala + val rule = new domain2.Rule + rule.setRuleNm("customer_before_company") + rule.setRuleValue("customer_since_date < '1956-11-01'") + rule.setPerc(1.0) + rule.setPoints(1) + rule.setIsActive(1) + rule.setUserNm("Kirk") + rule.setDataset("ATM_CUSTOMER3") + Util.addRule(rule=rule) +``` + +Now let's chain together the remaining 2 items that were part of our original requirement. Note that Owl has 6 additional ML DQ features that we did not turn on in this case. + +```scala +val owl = Util.OwlContext(df, atmCustFile, props) + +// first register with catalog if not registered +owl.register(props) + +// Check if dataframe matches the source file 'atm_cust_file' +val source = owl.validateSrcDF +if (source.count() > 1) { + // create service now ticket and exit with fail based on not matching to original file +} + +owl.addAdHocRule(rule) +val ruleBreaks = owl.rulesDF +if (ruleBreaks.count() > 1) { + if (ruleBreaks.where($"score" > 5).count > 1) { + // create service now ticket and exit with fail based on rules + } +} + +val outliers = owl.outliersDF +if (outliers.where($"confidence" < 10).count > 3) { + // Owl email Alert to business group for attention + // where 3 outliers have a confidence below 10 +} +``` + +### Ingesting Intraday Files + +Here we illustrate an example of how to work with files when using Owl programmatically. This can be implemented in both a Notebook setting and in your own codebase. + +```scala + /////////////////////////////////////////////////////////////////////////// + // USE CASE - Ingesting Intraday Files // + /////////////////////////////////////////////////////////////////////////// + + // Part of your pipeline includes the ingestion of files that have the date + // and hour encoded in the file name. How do you process those files using Owl? + // + // Format: ___.csv + // + // Build up a data structure containg the files you want to process (here we + // just use a simple list, but you may want to be pulling from a pubsub + // queue, AWS bucket, etc...). Here we just use a simple file list of 6 + // hours of trade position data. + val position_files = List( + new File(getClass.getResource("/position_file_2019_11_03_08.csv").getPath), + new File(getClass.getResource("/position_file_2019_11_03_09.csv").getPath), + new File(getClass.getResource("/position_file_2019_11_03_10.csv").getPath), + new File(getClass.getResource("/position_file_2019_11_03_11.csv").getPath), + new File(getClass.getResource("/position_file_2019_11_03_12.csv").getPath), + new File(getClass.getResource("/position_file_2019_11_03_13.csv").getPath), + new File(getClass.getResource("/position_file_2019_11_03_14.csv").getPath)) + + // Create your spark session. + val spark = SparkSession.builder + .master("local") + .appName("test") + .getOrCreate() + + // Configure Owl. + val opt = new OwlOptions + opt.dataset = "positions" + opt.load.delimiter = "," + opt.spark.master = "local[1]" + opt.dupe.on = true + opt.dupe.include = Array("ticker", "cid") + opt.outlier.on = true + opt.outlier.key = Array("cid") + opt.outlier.timeBin = TimeBin.HOUR + // Customize this to only process a subset of the data. + opt.load.fileQuery = "select * from dataset" + + position_files.foreach { file: File => + // Tell Owl where to find the file. + opt.load.filePath = file.getPath + + // Parse the filename to construct the run date (-rd) that will be passed + // to Owl. + val name = file.getName.split('.').head + val parts = name.split("_") + val date = parts.slice(2, 5).mkString("-") + val hour = parts.takeRight(1).head + + // Must be in format 'yyyy-MM-dd' or 'yyyy-MM-dd HH:mm'. + val rd = s"${date} ${hour}" + + // Tell Owl to process data + opt.runId = rd + + // Create a DataFrame from the file. + val df = OwlUtils.load(opt.load.filePath, opt.load.delimiter, spark) + + // Instantiate an OwlContext with the dataframe and our custom configuration. + val owl = OwlUtils.OwlContext(df, spark, opt) + + // Make sure Owl has catalogued the dataset. + owl.register(opt) + + // Let Owl do the rest! + owl.owlCheck() + + } +``` + +![](<../../.gitbook/assets/image (13).png>) + +### All Pipeline Activities in 1 Line + +For brevity and convenience Owl allows a DF to be loaded in the constructor and in 1 line run all 9 dimensions of data quality "owl.owlcheck". To adjust the DQ dimensions you simply set the properties in the props object. + +```scala +val owl = Util.OwlContext(df, atmCustFile, props) +owl.owlCheck +``` + +#### Example of some common property settings + +```scala +val props = new Props() +props.filePath = s"${filePath}/atm_customer_${rd.replace("-","_")}.csv" +props.runId = rd +props.dateCol = "OWL_RUN_ID" +props.dataset = "ATM_CUSTOMER3" +props.del = "," +props.datasetSafety = false +props.calculateBoundaries = true +props.fileLookBack = true +props.timeBin = "DAY" + +// outlier, missing records +props.dl = true +props.dlKey = "customer_id" +props.dlLb = 4 + +// pattern mining +props.freqPatternMiningByKey = true +props.fpgKey = "customer_id" +props.fpgLookback = 4 +props.fpgDateCol = "OWL_RUN_ID" +props.fpgCols = "card_number,first_name,last_name,checking_savings" +props.fpgLowFreq = true + +// validate Src +props.validateSrc = true +props.valSrcKey = "customer_id" + +// fuzzy match +props.isDupe = true +props.dupeCutOff = 88 +props.depth = 3 +props.dupeExcludeCols = "customer_id,card_number,customer_since_date,OWL_RUN_ID" +``` + +![](../../.gitbook/assets/owl-spark-dq=pipeline.png) + +### Using Notebooks to build DQ Pipelines + +For examples on how to do this, see our Notebook repository below. + +{% embed url="https://github.com/kirkhas/owl-notebooks" %} + +## diff --git a/dq-job-examples/owlcheck-spark/README.md b/dq-job-examples/owlcheck-spark/README.md new file mode 100644 index 00000000..aa19be63 --- /dev/null +++ b/dq-job-examples/owlcheck-spark/README.md @@ -0,0 +1,68 @@ +# Batch + +### Scale + Data Science + +Where Scale meets Data Science. Scale linearly with your data by adding executors and/or memory + +``` +-f "file:///Users/home/salary_data.csv" \ +-d "," \ +-rd "2018-01-08" \ +-ds "salary_data" +-numexecutors 2 \ +-executormemory 2g +``` + +### Yarn Master + +If Owl is run on an edge node on a popular hadoop distribution such as HDP, CDH, EMR it will automatically register the jobs with Yarn Resource Manager. + +### Spark Master + +Owl can also run using spark master by using the -master input and passing in spark:url + +### Spark Standalone + +Owl can run in standalone most but naturally will not distribute the processing beyond the hardware it was activated on. + +| Options | Description | +| -------------- | ------------------------------------------------------------------------- | +| deploymode | spark deploymode option | +| drivermemory | driver memory example 3G for local space | +| executorcores | spark executor cores | +| executormemory | spark executor memory option example 3G | +| master | overrides local\[\*], i.e. spark://myhost:7077, yarn-client, yarn-cluster | +| sparkprinc | kerberos principal name ex: owl@OWL.COM | + +### Use Spark-Submit directly bypassing OwlCheck + +```bash +spark-submit \ +--driver-class-path /opt/owl/drivers/postgres42/postgresql-42.2.4.jar \ +--driver-library-path /opt/owl/drivers/postgres42/postgresql-42.2.4.jar \ +--driver-memory 3g --num-executors 2 --executor-memory 1g \ +--master spark://Kirks-MBP.home:7077 \ +--class com.owl.core.cli.OwlCheck /opt/owl/bin/owl-core-trunk-jar-with-dependencies.jar \ +-u user -p pass -c jdbc:postgresql://xyz.chzid9w0hpyi.us-east-1.rds.amazonaws.com/postgres \ +-ds accounts -rd 2019-05-05 -dssafeoff -q "select * from accounts" +-driver org.postgresql.Driver -lib /opt/owl/drivers/postgres42/ +``` + +### Parallel JDBC Spark-Submit + +```bash +spark-submit \ +--driver-class-path /opt/owl/drivers/postgres42/postgresql-42.2.4.jar \ +--driver-library-path /opt/owl/drivers/postgres42/postgresql-42.2.4.jar \ +--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:///opt/owl/config/log4j-TRACE.properties \ +--conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:///opt/owl/config/log4j-TRACE.properties \ +--files /opt/owl/config/log4j-TRACE.properties \ +--driver-memory 2g --num-executors 2 --executor-memory 1g --master spark://Kirks-MBP.home:7077 \ +--class com.owl.core.cli.OwlCheck /opt/owl/bin/owl-core-trunk-jar-with-dependencies.jar \ +-u us -p pass -c jdbc:postgresql://xyz.chzid9w0hpyi.us-east-1.rds.amazonaws.com/postgres \ +-ds aumdt -rd 2019-05-05 -dssafeoff -q "select * from aum_dt" \ +-driver org.postgresql.Driver -lib /opt/owl/drivers/postgres42/ \ +-connectionprops fetchsize=6000 -master spark://Kirks-MBP.home:7077 \ +-corroff -histoff -statsoff \ +-columnname updt_ts -numpartitions 4 -lowerbound 1557597987353 -upperbound 1557597999947 +``` diff --git a/dq-job-examples/owlcheck-spark/job-files.md b/dq-job-examples/owlcheck-spark/job-files.md new file mode 100644 index 00000000..6f46ec63 --- /dev/null +++ b/dq-job-examples/owlcheck-spark/job-files.md @@ -0,0 +1,277 @@ +# DQ Job Files + +For example, a large file `transaction_2021-01-01.csv` might contain the following transaction data with two transaction per day spanning all of January. + +| transaction\_id | account\_id | date | amount | +| --------------- | ----------- | ---------- | ------ | +| 1 | 1 | 2021-01-01 | 100 | +| 2 | 2 | 2021-01-01 | 120 | +| 3 | 1 | 2021-01-02 | 90 | +| 4 | 2 | 2021-01-02 | 115 | +| ... | | ... | ... | +| 61 | 1 | 2021-01-31 | 100 | +| 62 | 2 | 2021-01-31 | 999 | + +and this file might be located on the directory `~/customer/transaction-2021-01-01/`. + +```bash +~/customer + ├── transaction-2021-01-01 + │ └── transaction_2021-01-01.csv + ├── transaction-2021-02-01 + │ └── transaction_2021-02-01.csv + ... # folders for 2021-03-01 to 2021-11-01 ommitted + ├── transaction-2021-12-01 + │ └── transaction_2021-12-01.csv +``` + +Other folders with similar pattern may exist in your directory, such as `~/customer/transaction-2021-02-1`. Note that February data is located in a separate directory with a similar pattern for all the months of 2021. This dataset could similarly have 2 account IDs and 1 transaction per account per day (= 28 x 2 = 56 rows of data). For this example, let's assume this is the case for all the files. + +To run an Owlcheck on this single file containing multiple dates, you have the following choices. + +## Owlchecks with **file** + +**1. Run an Owlcheck on all the rows in a single file.** + +```bash +./owlcheck + -ds DQCheck_transactions_jan21 + -rd "2021-01-01" + -f "~/customer/transaction-2021-01-01/transaction_2021-01-01.csv" + -fq "select * from dataset" + ... # other relevant options +``` + +Here we assume that run date (`-rd`) is "2021-01-01" because it is currently January 1, 2021.\ +\ +The above command would lead to an Owlcheck on 62 rows of data spanning all of January 2021 from a single file located at `~/customer/transaction-2021-01-01/transaction_20210101.csv`. \ +\ +If you were to schedule a job to run this job monthly and next job ran on February 1st, 2021, then same DQ checks will be performed on the **same set of 62 rows** with same score as your Owlcheck run from January 1, 2021.\ +\ + For example, the follow-up scheduled job running on February 1st, 2021 would be: + +```bash +./owlcheck + -ds DQCheck_transactions_jan21 + -rd "2021-02-01" + -f "~/customer/transaction-2021-01-01/transaction_2021-01-01.csv" + -fq "select * from dataset" + ... # other relevant options +``` + +\ +This type of Owlcheck on a single file is suitable if you are verifying a static file `~/customer/transaction-2021-01-01/transaction_20210101.csv` that does not change over time and expect the score to be the same every run. Hence, it is suggested to name the dataset that reflect this, such as `DQCheck_transaction_jan21` to reflect the idea that this dataset is checking the Data Quality of transaction table containing January 2021 data. Similar Owlcheck for February 2021 data would then be a separate and independent dataset named `DQCheck_transaction_feb21`\ +\ +This type of Owlcheck can also be used if `~/customer/transaction-2021-01-01/transaction_20210101.csv` is changing (the rows are changing values or new rows are being added) and want to detect data quality changes. Transaction file doesn't fit with this scenario, but the idea is that the above command specifies Data Quality Owlchecks on the entirety of the file. The run date is a date that you choose to assign for that Owlcheck. It is conventional to have one-to-one mapping between run date and the date corresponding to the date that DQ checks are being performed. Run date does not have to match with the data underlying the file. + +**2. Run an Owlcheck on subset of rows from a single file** + +The single file contains daily data for January of 2021. To run Data Quality checks on January 1st, January 2nd, ... , and January 31st, you need to run 31 Owlchecks, each with subset of rows from the file. Note the `where` clause in `-fq` matching with the run date `-rd` + +```bash +./owlcheck + -ds DQCheck_transactions_jan21 + -rd "2021-01-01" + -f "~/customer/transaction-2021-01-01/transaction_2021-01-01.csv" + -fq "select * from dataset where date = '2021-01-01'" + ... # other relevant options + +./owlcheck + -ds DQCheck_transactions_jan21 + -rd "2021-01-02" + -f "~/customer/transaction-2021-01-01/transaction_2021-01-01.csv" + -fq "select * from dataset where date = '2021-01-02'" + ... # other relevant options + +... # Owlchecks for -rd 2021-01-03 to 2021-01-30 ommitted + +./owlcheck + -ds DQCheck_transactions_jan21 + -rd "2021-01-31" + -f "~/customer/transaction-2021-01-01/transaction_2021-01-01.csv" + -fq "select * from dataset where date = '2021-01-31'" + ... # other relevant options + +``` + +By using the same dataset name `-ds`, all 31 Owlchecks will appear under one dataset `DQCheck_transaction_jan21` in the Hoot page. + +A convenient way to parameterize this run date is to use `${rd}` in the query. + +```bash +./owlcheck + -ds DQCheck_transactions_jan21 + -rd "2021-01-01" + -f "~/customer/transaction-2021-01-01/transaction_2021-01-01.csv" + -fq "select * from dataset where date = '${rd}'" + ... # other relevant options + +``` + +A daily scheduled job starting on January 1st, 2021 to January 31, 2021 will automatically replace the `${rd}` with "2021-01-01", "2021-01-02", ... , and "2021-01-31" for the respective run date. + +**3. Run an Owlcheck on subset of rows from a single file with day lookback** + +For certain core components like **Outlier**, a set of rows corresponding to historical training data can be used to establish a baseline. For example, the row with `transaction_id` 62 has amount of 999. This looks like an outlier that we want to catch. This value of 999 seems to be an outlier because past transaction amounts for `account_id`2 are in the 100s range. We can use historical data from January 15th to January 30th and use that info to see if January 31st data contains any outliers. \ +\ +In this scenario, our single file `~/customer/transaction-2021-01-01/transaction_2021-01-01.csv` contains such historical data because that file contains all the data for all of January. How do we use the same file for both current data (January 31st) and historical (January 15th to January 30th) data? You do not have to split the files into two. You can simply do exactly what you would do for Owlcheck on "2021-01-31" with a `-fullfile` flag. The `-fullfile` flag tells the Owlcheck that "the file in `-f` contains the historical data. Construct a query and subset those rows for me". + +```bash +./owlcheck + -ds DQCheck_transactions_jan21 + -rd "2021-01-31" + -f "~/customer/transaction-2021-01-01/transaction_2021-01-01.csv" + -fq "select * from dataset where date = '2021-01-31'" + -fullfile + # outlier options + -dc "date" + -dl + -tbin "DAY" # look back time bin is day + -dllb 15 # look back up to 15 days + ... # other relevant options +``` + +## Owlchecks with _multiple files_ + +**4. Run an Owlcheck on a single file with lookback using series of file** + +Recall our folder structure: + +```bash +~/customer + ├── transaction-2021-01-01 + │ └── transaction_2021-01-01.csv + ├── transaction-2021-02-01 + │ └── transaction_2021-02-01.csv + ... # ommitted for space + ├── transaction-2021-12-01 + │ └── transaction_2021-12-01.csv +``` + +If we want to run an Owlcheck for December of 2021 and use July of 2021 to November of 2021 as our historical training dataset, how can we load _multiple_ _files_? Just like how `-fullfile` provides a convenient way to create historical training dataset on a single file, `-fllb` (file lookback) provides a convenient way to load _series of files_ with patterns while still pointing to the target file (December file) in `-f` + +```bash +./owlcheck + -ds DQCheck_transactions_dec21 + -rd "2021-12-01" + -f "~/customer/transaction-2021-12-01/transaction_2021-12-01.csv" + -fq "select * from dataset" + -fllb + # outlier options + -dc "date" + -dl + -tbin "MONTH" + -dllb 5 # look back up to 5 months + ... # other relevant options +``` + +One caveat to this `-fllb` method __ is that the Owlcheck history must be "primed" first so that the OwlDQ knows the file path of the past series of files. In fact, `-fllb` does not use the file path provided in `-f` and loads different files from different folders. _It relies on the Owlcheck history_ under the same `-ds` name. `-fllb` means lookback up to N number of past consecutive Owlchecks. For each of those past Owlcheck, look up the file path `-f` used in the past and follow those paths. The number N is determined by the maximum number of lookbacks from Outlier (`-dllb`) and Patterns (`-fpglb` ). In the Owlcheck above, because `-dllb 5` is provided along with `-fllb`, it means "Look up 5 past owlchecks and load those files as historical dataset".\ +\ +In summary, in order to run an Owlcheck on "2021-12-01" and have that Owlcheck for that date "look up" the files in \ + `~/customer/transaction-2021-07-01/transaction_2021-07-01.csv` ,\ + `~/customer/transaction-2021-08-01/transaction_2021-08-01.csv` ,\ + `~/customer/transaction-2021-09-01/transaction_2021-09-01.csv` ,\ + `~/customer/transaction-2021-10-01/transaction_2021-10-01.csv` , and\ + `~/customer/transaction-2021-11-01/transaction_2021-11-01.csv` , \ +\ +you need to have ran Owlchecks for "2021-07-01", "2021-08-01", ... , and "2021-11-01" under the same dataset name. Therefore, it would be more logical, best-practice is to name the dataset `-ds DQCheck_transaction_2021` and run series of monthly owlchecks up to "2021-12-01" (but the name of the dataset is up to you)/ + +```bash +# Prime past Owlchecks so that "2021-12-01" knows the file path of past months +./owlcheck + -ds DQCheck_transactions_2021 + -rd "2021-07-01" + -f "~/customer/transaction-2021-07-01/transaction_2021-07-01.csv" + -fq "select * from dataset" + +./owlcheck + -ds DQCheck_transactions_2021 + -rd "2021-08-01" + -f "~/customer/transaction-2021-08-01/transaction_2021-08-01.csv" + -fq "select * from dataset" + + +./owlcheck + -ds DQCheck_transactions_2021 + -rd "2021-09-01" + -f "~/customer/transaction-2021-08-01/transaction_2021-09-01.csv" + -fq "select * from dataset" + +./owlcheck + -ds DQCheck_transactions_2021 + -rd "2021-10-01" + -f "~/customer/transaction-2021-08-01/transaction_2021-10-01.csv" + -fq "select * from dataset" + +./owlcheck + -ds DQCheck_transactions_2021 + -rd "2021-11-01" + -f "~/customer/transaction-2021-11-01/transaction_2021-11-01.csv" + -fq "select * from dataset" + +# Priming 5 past Owlchecks complete. Now run the 2021-12-01 +./owlcheck + -ds DQCheck_transactions_2021 + -rd "2021-12-01" + -f "~/customer/transaction-2021-12-01/transaction_202-11-201.csv" + -fq "select * from dataset" + -fllb + # outlier options + -dc "date" + -dl + -tbin "MONTH" + -dllb 5 # look back up to 5 months + ... # other relevant options +``` + +In this scenario, since the folder paths have a pattern, we can use `-br` for priming in one command instead of writing 5 Owlcheck commands. The flag `-br` runs Owlchecks consecutively from the past and increments by monthly if `-tbin "MONTH"` (by default `-tbin DAY`so the default behavior is to increment daily). The different folder paths on each past consecutive run dates are replaced with `${rd}`.\ +\ +The below command is identical to the above + +```bash +# Prime past Owlchecks so that "2021-12-01" knows the file path of past months +./owlcheck + -ds DQCheck_transactions_2021 + -rd "2021-12-01" + -f "~/customer/transaction-${rd}/transaction_${rd}.csv" + -fq "select * from dataset" + -br 5 # run 5 runs to in the past + -tbin "MONTH" # <-- required since we want a MONTHLY backrun. Default "DAY" + +# Priming complete. Now run the 2021-12-01 +./owlcheck + -ds DQCheck_transactions_2021 + -rd "2021-12-01" + -f "~/customer/transaction-2021-12-01/transaction_2021-12-01.csv" + -fq "select * from dataset" + -fllb + # outlier options + -dc "date" + -dl + -tbin "MONTH" + -dllb 5 # look back up to 5 months + ... # other relevant options +``` + +This pattern is designed so that a single Owlcheck command can be scheduled and `${rd}` be used to replace the folder & file path. Your `~/customer` folder could contain transactions for all the years, spanning all the way back to 1992 and into the future like so: + +```bash +~/customer + + ├── transaction-1992-01-01 + │ └── transaction_1992-01-01.csv + ├── transaction-1992-01-01 + │ └── transaction_1992-01-01.csv + ... # ommitted for space + ├── transaction-2021-01-01 + │ └── transaction_2021-01-01.csv + ├── transaction-2021-02-01 + │ └── transaction_2021-02-01.csv + ... # ommitted for space + ├── transaction-2021-12-01 + │ └── transaction_2021-12-01.csv + ... # hasn't happened yet! + +``` + +In this scenario, a monthly scheduled job would get rid of the need to "prime" the Owcheck history, since your past scheduled jobs would have already ran the past Owlchecks. diff --git a/dq-job-examples/owlcheck-spark/job-hive.md b/dq-job-examples/owlcheck-spark/job-hive.md new file mode 100644 index 00000000..3a141fba --- /dev/null +++ b/dq-job-examples/owlcheck-spark/job-hive.md @@ -0,0 +1,108 @@ +# DQ Job Hive + + + +Run a Data Quality check on a Hive table. Use the -hive flag for a native connection via the HCat, this does not require a JDBC connection and is optimized for distributed speed and scale. + +### Hive Native, no JDBC Connection + +Open source platforms like HDP, EMR and CDH use well known standards and because of this Owl can take advantage of things like HCat and it removes the need for JDBC connection details as well as offers optimum data read speeds. Owl recommends and supports this with the -hive flag. + +```bash +./owlcheck -ds hive_table -rd 2019-03-13 \ +-q "select * from hive_table" -hive +``` + +Example output. A hoot is a valid JSON response + +```bash +{ + "dataset": "hive_table", + "runId": "2019-02-03", + "score": 100, + "behaviorScore": 0, + "rows": 477261, + "prettyPrint": true +} +``` + +### Hive JDBC + +1. You need to use the hive JDBC driver, commonly org.apache.hive.HiveDriver +2. You need to locate your driver JDBC Jar with the version that came with your EMR, HDP or CDH + 1. This jar is commonly found on an edge node under /opt/hdp/libs/hive/hive-jdbc.jar etc... + +```bash +./owlcheck -rd 2019-06-07 -ds hive_table \ +-u -p -q "select * from table" \ +-c "jdbc:hive2://:10000/default" \ +-driver org.apache.hive.HiveDriver \ +-lib /opt/owl/drivers/hive/ \ +-master yarn -deploymode client +``` + +#### HDP Driver - org.apache.hive.HiveDriver + +#### CDH Driver - com.cloudera.hive.jdbc41.Datasource + +For CDH all the drivers are packaged under, HiveJDBC41\_cdhversion.zip + +### Troubleshooting + +A common JDBC connection is hive.resultset.use.unique.column.names=false + +This can be added directly to the JDBC connection url string or to the driver properties section + +![](<../../.gitbook/assets/image (60).png>) + + + + + +Test your hive connection via beeline to make sure it is correct before going further. + +```bash +beeline -u 'jdbc:hive2://:10000/default;principal=hive/cdh-instance1.us-east1-b.c.company-hadoop-cdh.internal@CW.COM;useSSL=true' -d org.apache.hive.jdbc.HiveDriver +``` + +#### Kerberos Example + +```bash +jdbc:hive2://:10000/default;principal=hive/cdh-instance1.us-east1-b.c.company-hadoop-cdh.internal@CW.COM;useSSL=true +``` + +### Connecting Owl WebApp to Hive JDBC + +Notice the driver properties for kerberos and principals. + +![](<../../.gitbook/assets/Screen Shot 2019-05-29 at 6.44.50 PM.png>) + +In very rare cases where you can't get the jar files to connect properly one workaround is to add this to the owl-web startup script + +``` +$JAVA_HOME/bin/java -Dloader.path=lib,/home/danielrice/owl/drivers/hive/ \ +-DowlAppender=owlRollingFile \ +-DowlLogFile=owl-web -Dlog4j.configurationFile=file://$INSTALL_PATH/config/log4j2.xml \ +$HBASE_KERBEROS -jar $owlweb $ZKHOST_KER \ +--logging.level.org.springframework=INFO $TIMEOUT \ +--server.session.timeout=$TIMEOUT \ +--server.port=9001 > $LOG_PATH/owl-web-app.out 2>&1 & echo $! >$INSTALL_PATH/pids/owl-web.pid +``` + +### Class Not Found apache or client or log4j etc... + +Any class not found error means that you do not have the "standalone-jar" or you do not have all the jars needed for the driver. + +#### Hive JDBC Jars + +It is common for Hive to need a lot of .jar files to complete the driver setup. + +![](../../.gitbook/assets/cdh-jars-hive-jdbc.png) + +### Java jar cmds + +Sometimes it is helpful to look inside the jar and make sure it has all the needed files. + +``` +jar -tvf hive-jdbc.jar +``` diff --git a/dq-job-examples/owlcheck-spark/job-jdbc.md b/dq-job-examples/owlcheck-spark/job-jdbc.md new file mode 100644 index 00000000..a514dc3d --- /dev/null +++ b/dq-job-examples/owlcheck-spark/job-jdbc.md @@ -0,0 +1,61 @@ +# DQ Job JDBC + +Connect to any database via JDBC. + +```bash +-q "select * from lake.stock_eod where date = '2017-01-20' " \ +-u username -p password \ +-c "jdbc:mysql://instance.chzid9w0hpyi.us-east-1.rds.amazonaws.com:3306" \ +-rd "2017-01-20" \ +-dc "date" \ +-ds "stocks" \ +-driver com.mysql.jdbc.Driver \ +-lib "/home/ec2-user/owl/drivers/mysql/" +``` + +![](../../.gitbook/assets/owl-connection.png) + +### Password Manager + +You can configure Owl to call a script file to retrieve a password from a password manager vault or other storage container. The customer is responsible for generating a script to pull just the password and Owl will use that value dynamically when the connection is needed for the UI or when kicking off an Owlcheck. + +![](<../../.gitbook/assets/Screen Shot 2021-02-10 at 8.55.23 PM.png>) + +In the connection dialog, select Password Manager from the Auth Type dropdown, and supply a user name. The script is the path to the .sh script on the machine where the web application is running, and the user account that runs Owl-web should be allowed to execute the script. You can either use the optional parameters or pass any parameters your script needs directly inline on the Script value. + +### Fetch Size + +It is important to consider the drivers fetch size when loading greater than 1 Million rows across the network. Owl allows you to set this driver property in the WebApp but this is only for web interaction therefore "fetchsize" will not help here. Owl also allows fetchsize in the OwlCheck by passing in a connection property. + +#### CMD line + +``` +-connectionprops "fetchsize=3000" +``` + +#### Notebook + +``` +props.connectionProps.put("fetchsize", "3000") +``` + +## Parallel JDBC + +For greater performance or moving large datasets across a network Owl supports parallel JDBC, which can be enabled by passing `numpartitions` to Owlcheck. This can be a 2-5X improvement in many cases. + +```bash +-lib "/opt/owl/drivers/mysql8/" +-cxn mysql +-q "select * from lake.nyse where trade_date = '${rd}' " +-rd 2018-01-01 +-ds nyse +-columnname volume +-numpartitions 4 +-lowerbound "0" +-upperbound "5000000000" +-usesql +``` + +Owl also supports auto parallelization, which will configure the `numPartitions` parameter for you based on the size of your data. This is enabled in the UI when you create a dataset using the Owlcheck wizard. + +![](<../../.gitbook/assets/Screen Shot 2019-10-17 at 4.38.04 PM.png>) diff --git a/dq-job-examples/owlcheck-spark/job-json.md b/dq-job-examples/owlcheck-spark/job-json.md new file mode 100644 index 00000000..58614ecf --- /dev/null +++ b/dq-job-examples/owlcheck-spark/job-json.md @@ -0,0 +1,99 @@ +# DQ Job JSON + +## Files + +Run against a file using -json. Additionally, options are available for -flatten and -multiline. This is helpful for nested and various formats. + +``` +-ds json_file_example \ +-f s3a://bucket_name/file.json \ +-h instance.us-east4-c.c.owl-node.internal:5432/postgres \ +-master spark://instance.us-east4-c.c.owl-node.internal:7077 \ +-json \ +-flatten \ +-multiline +``` + +{% hint style="info" %} +Automatic flattening will infer schema and explode all structs, arrays, and map types. +{% endhint %} + +## Using Spark SQL + +```bash +-ds public.json_sample \ +-lib "/opt/owl/drivers/postgres/" \ +-h instance.us-east4-c.c.owl-node.internal:5432/postgres \ +-master spark://instance.us-east4-c.c.owl-node.internal:7077 +-q "select * from public.jason" +-rd "2021-01-17" +-driver "org.postgresql.Driver" +-cxn postgres-gcp +-fq "select \ +get_json_object(col_3, '$.data._customer_name') AS `data_customer_name` , \ +get_json_object(col_3, '$.data._active_customer') AS `data_active_customer` , \ +from dataset " + +``` + +{% hint style="info" %} +Pass in the path to Owls' -fq parameter. This is great for mixed data types within a database. For example, if you store JSON data as a string or a blob among other data. +{% endhint %} + +```bash +// Flatten +val colArr = new JsonReader().flattenSchema(df.schema) +colArr.foreach(x => println(x)) +``` + +{% hint style="success" %} +This Owl utility will traverse the entire schema and print the proper get JSON object spark sql strings. You can use this instead of typing each query statement into the command line -fq parameter as seen above. +{% endhint %} + +## Using Owl Libraries + +{% code title="" %} +```bash +import com.owl.common.options._ +import com.owl.core.util.OwlUtils +import com.owl.core.activity2.JsonReader + +val connProps = Map ( + "driver" -> "org.postgresql.Driver", + "user" -> "user", + "password" -> "password", + "url" -> "jdbc:postgresql://10.173.0.14:5432/postgres", + "dbtable" -> "public.data" +) + +// Spark +var rdd = spark.read.format("jdbc").options(connProps).load.select($"col_name").map(x=>x.toString()).rdd +var df = spark.read.json(rdd) + +// Flatten +val colArr = new JsonReader().flattenSchema(df.schema) +val flatJson = df.select(colArr: _*) +flatJson.cache.count + +// Opts +val dataset = "json_example" +val runId = s"2021-01-14" +val opt = new OwlOptions() +opt.dataset = dataset +opt.runId = runId +opt.datasetSafeOff = true + +// Owlcheck +OwlUtils.resetDataSource("instance.us-east4-c.c.owl-node.internal","5432/postgres","user","pass", spark) +val owl = OwlUtils.OwlContext(flatJson, opt) +owl.register(opt) +owl.owlCheck +``` +{% endcode %} + +{% hint style="info" %} +### JsonReader() + +This uses Owl's JsonReader to do the heavy lifting. +{% endhint %} + diff --git a/dq-job-examples/owlcheck-spark/job-mysql.md b/dq-job-examples/owlcheck-spark/job-mysql.md new file mode 100644 index 00000000..3695a903 --- /dev/null +++ b/dq-job-examples/owlcheck-spark/job-mysql.md @@ -0,0 +1,22 @@ +# DQ Job MySql + +### Video Tutorial (MySQL) + +Add automatic data quality to any database in 60 seconds. This example shows a single table being selected for DQ, however Owl also provides the ability to scan all schemas and tables at once. + +{% embed url="https://vimeo.com/372286610" %} + + + +Connect to any database using JDBC. Mysql example below. + +```bash +-q "select * from lake.stock_eod where date = '2017-01-20' " \ +-u username -p password \ +-c "jdbc:mysql://owldatalake.chzid9w0hpyi.us-east-1.rds.amazonaws.com:3306" \ +-rd "2017-01-20" \ +-dc "date" \ +-ds "stocks" \ +-driver com.mysql.jdbc.Driver \ +-lib "/home/ec2-user/owl/drivers/mysql/" +``` diff --git a/dq-job-examples/owlcheck-spark/owlcheck-bigquery.md b/dq-job-examples/owlcheck-spark/owlcheck-bigquery.md new file mode 100644 index 00000000..68584113 --- /dev/null +++ b/dq-job-examples/owlcheck-spark/owlcheck-bigquery.md @@ -0,0 +1,107 @@ +# DQ Job BigQuery + +## Example CMD Line + +``` +-lib "/opt/owl/drivers/bigquery/bigquery/core/" \ +-h :5432/postgres \ +-master spark://:7077 \ +-ds samples.loan_customer \ +-deploymode client \ +-q "select * from samples.loan_customer" \ +-rd "2021-08-02" \ +-driver "com.simba.googlebigquery.jdbc42.Driver" \ +-cxn BigQuery +``` + +### Steps for the BigQuery Connection + +1. **We would use this Simba driver**: com.simba.googlebigquery.jdbc42.Driver +2. **We would make an owl-gcp.json** (your org auth key in JSON format) +3. **We would create a JDBC connection** (for example only do not use this JDBC URL):\ + jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=;OAuthType=0;OAuthServiceAcctEmail=<1234567890>[-compute@developer.gserviceaccount.com;OAuthPvtKeyPath=/opt/ext/owl-gcp.json;Timeout=86400](mailto:-compute@developer.gserviceaccount.com;OAuthPvtKeyPath=/opt/ext/owl-gcp.json;Timeout=86400) +4. **Requires a path to a JSON file** that contains the service account for authorization. That same file is provided to the Spark session to make a direct to storage connection for maximum parallelism once Core fires up.” + +The above and explained there are actually a number of others steps which must be performed to achieve success: + +1. **Password for the BigQuery Connector form in Collibra DQ must be a base64 encoded string created from the json file (see step 3. above)** and input as password. For example:\ + `base64 your_json.json -w 0`\ + or\ + `cat your_json.json | base64 -w 0` +2. **Check that this JARs exists and is on the path of the Collibra DQ Web UI server** (eg. \/owl/drivers/bigquery/core). Look at your driver directory location which contains this BigQuery JAR: spark-bigquery\_2.12-0.18.1.jar +3. **Make sure there are all the needed JARs present in \/owl/drivers/bigquery/:**\ + ****_animal-sniffer-annotations-1.19.jar_\ + _google-api-services-bigquery-v2-rev20201030-1.30.10.jar_\ + _grpc-google-cloud-bigquerystorage-v1beta1-0.106.4.jar_\ + _listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar_\ + _annotations-4.1.1.4.jar_\ + _google-auth-library-credentials-0.22.0.jar_\ + _grpc-google-cloud-bigquerystorage-v1beta2-0.106.4.jar_\ + _opencensus-api-0.24.0.jar_\ + _api-common-1.10.1.jar_\ + _google-auth-library-oauth2-http-0.22.0.jar_\ + _grpc-grpclb-1.33.1.jar_\ + _opencensus-contrib-http-util-0.24.0.jar_\ + _auto-value-annotations-1.7.4.jar_\ + _GoogleBigQueryJDBC42.jar_\ + _grpc-netty-shaded-1.33.1.jar_\ + _perfmark-api-0.19.0.jar_\ + _avro-1.10.0.jar_\ + _google-cloud-bigquery-1.125.0.jar_\ + _grpc-protobuf-1.33.1.jar_\ + _protobuf-java-3.13.0.jar_\ + _checker-compat-qual-2.5.5.jar_\ + _google-cloud-bigquerystorage-1.6.4.jar_\ + _grpc-protobuf-lite-1.33.1.jar_\ + _protobuf-java-util-3.13.0.jar_\ + _commons-codec-1.11.jar_\ + _google-cloud-core-1.93.10.jar_\ + _grpc-stub-1.33.1.jar_\ + _proto-google-cloud-bigquerystorage-v1-1.6.4.jar_\ + _commons-compress-1.20.jar_\ + _google-cloud-core-http-1.93.10.jar_\ + _gson-2.8.6.jar_\ + _proto-google-cloud-bigquerystorage-v1alpha2-0.106.4.jar_\ + _commons-lang3-3.5.jar_\ + _google-http-client-1.38.0.jar_\ + _guava-23.0.jar_\ + _proto-google-cloud-bigquerystorage-v1beta1-0.106.4.jar_\ + _commons-logging-1.2.jar_\ + _google-http-client-apache-v2-1.38.0.jar_\ + _httpclient-4.5.13.jar_\ + _proto-google-cloud-bigquerystorage-v1beta2-0.106.4.jar_\ + _conscrypt-openjdk-uber-2.5.1.jar_\ + _google-http-client-appengine-1.38.0.jar_\ + _httpcore-4.4.13.jar_\ + _proto-google-common-protos-2.0.1.jar_\ + _core_\ + _google-http-client-jackson2-1.38.0.jar_\ + _j2objc-annotations-1.3.jar_\ + _proto-google-iam-v1-1.0.3.jar_\ + _error\_prone\_annotations-2.4.0.jar_\ + _google-oauth-client-1.31.1.jar_\ + _jackson-annotations-2.11.0.jar_\ + _grpc-alts-1.33.1.jar_\ + _jackson-core-2.11.3.jar_\ + _slf4j-api-1.7.30.jar_\ + _failureaccess-1.0.1.jar_\ + _grpc-api-1.33.1.jar_\ + _jackson-databind-2.11.0.jar_\ + _gax-1.60.0.jar_\ + _grpc-auth-1.33.1.jar_\ + _javax.annotation-api-1.3.2.jar_\ + _threetenbp-1.5.0.jar_\ + _gax-grpc-1.60.0.jar_\ + _grpc-context-1.33.1.jar_\ + _joda-time-2.10.1.jar_\ + _gax-httpjson-0.77.0.jar_\ + _grpc-core-1.33.1.jar_\ + _json-20200518.jar_\ + _google-api-client-1.31.1.jar_\ + _grpc-google-cloud-bigquerystorage-v1-1.6.4.jar_\ + _jsr305-3.0.2.jar_ +4. You may get a CLASSPATH conflict regarding the JAR files. +5. Make sure the BigQuery connector Scala version matches your Spark Scala version.[![02%20PM](https://discourse-static.influitive.net/uploads/db\_033c9cc6\_3cea\_4623\_b4a8\_52ebc3f9e8a1/optimized/2X/d/dfca73373275afb5f063f192a3aa7105caa76bd8\_2\_286x500.png)](https://discourse-static.influitive.net/uploads/db\_033c9cc6\_3cea\_4623\_b4a8\_52ebc3f9e8a1/original/2X/d/dfca73373275afb5f063f192a3aa7105caa76bd8.png) + + + diff --git a/dq-job-examples/owlcheck-spark/owlcheck-databricks.md b/dq-job-examples/owlcheck-spark/owlcheck-databricks.md new file mode 100644 index 00000000..d35a571b --- /dev/null +++ b/dq-job-examples/owlcheck-spark/owlcheck-databricks.md @@ -0,0 +1,34 @@ +# DQ Job Databricks + +### Lake vs Swamp + +The difference between a business-critical lake and a swamp is _quality_. The accuracy and cleanliness of data is directly proportional to the quality of insights end-users will derive. Data lakes that gain broad adoption have strong governance programs. The challenge is, adding a DQ program typically takes 6-12 months but the project never really ends due to the volume, variety and velocity of incoming data. OwlDQ uses autoML so solve this problem. OwlDQ constantly monitors the lake with native integration and unlimited scale. Use OwlDQ to generate the equivalent of 10K rules, while continuously adapting to the natural variance in your data. When erroneous data enters your lake OwlDQ will alert the data steward and provide a rich visual displaying the break records and explainable AI describing the issue. OwlDQ's approach is to learn from data and become incrementally smarter each day to ensure a statistically defensible DQ program. + +### Native Integration with Delta Lake (Databricks) + +Out of the box OwlDQ comes with a connection template for Databricks. To connect, simply paste in your _username_, _password_ and _connection URL_. + +![](../../.gitbook/assets/owl-databricks.png) + +### Explore Databricks Assets and Add DQ Checks + +Quickly explore DB assets that are cataloged in Delta Lake the same way you would any database (file tree explorer). Use OwlDQ wizard to add data quality to any Databricks table or file. Create a modern DQ program using machine learning in minutes. + +![](<../../.gitbook/assets/Screen Shot 2020-01-30 at 7.22.56 PM.png>) + +## 9 dimensions of Data Quality + +Use the wizard to apply Owl's autoML and predictive DQ features across all of your assets in Delta Lake. Click Scan button to put every table in Delta Lake under DQ management in 1 click. Owl can create a data quality program on all Delta Lake assets in a matter of hours. With traditional technologies this task used to require domain experts, rule writers and identification of critical elements. + +### Out of the Box DQ measures + +| DQ Dimension | Desc | +| ---------------- | ------------------------------------------ | +| Outliers | numeric and categorical outlier detection | +| Shapes | formatting and incorrect characters | +| Patterns | relationship probabilities | +| Correlations | strengths of relationships between columns | +| Duplicates | fuzzy and exact matching | +| Schema Evolution | schema drift | +| Rules | ability to add your own business rules | +| Source Matching | difference from source to target detection | diff --git a/dq-job-examples/owlcheck-spark/owlcheck-hdfs.md b/dq-job-examples/owlcheck-spark/owlcheck-hdfs.md new file mode 100644 index 00000000..68ae1095 --- /dev/null +++ b/dq-job-examples/owlcheck-spark/owlcheck-hdfs.md @@ -0,0 +1,16 @@ +# DQ Job HDFS + + + +Run Data Quality on a file in HDFS. Owl will automatically infer the schema and create an internal training model. + +``` +-f "hdfs:///demo/ssn_test2.csv" \ +-d "," \ +-rd "2018-01-08" \ +-ds "ssn_hdfs_file" \ +-master yarn \ +-deploymode cluster \ +-numexecutors 2 \ +-executormemory 2g +``` diff --git a/dq-job-examples/owlcheck-spark/owlcheck-mongodb.md b/dq-job-examples/owlcheck-spark/owlcheck-mongodb.md new file mode 100644 index 00000000..0d41c9a0 --- /dev/null +++ b/dq-job-examples/owlcheck-spark/owlcheck-mongodb.md @@ -0,0 +1,64 @@ +# DQ Job MongoDB + +## Browse MongoDB like any other relational database + +Using Owl's file tree explorer browse Mongo "collections" like "tables". Then use the wizard to create standard DQ scans. + +![](<../../.gitbook/assets/Screen Shot 2020-08-01 at 10.11.41 AM.png>) + +### CMD Line + +Copy paste-able cmdline example for simple spark submit job. + +```bash +-lib "/opt/owl/drivers/mongodb/" +-h localhost:5432/postgres +-master local[*] +-ds tpch.lineitem_7 +-br 10 -deploymode client +-q "select * from tpch.lineitem where l_shipdate between '${rd} 00:00:00.000+0000' +and '${rdEnd} 00:00:00.000+0000' " +-bhlb 10 -rd "1998-12-01" +-driver "mongodb.jdbc.MongoDriver" +-loglevel INFO -cxn MongoDB -rdEnd "1998-12-02" +``` + +## Drivers and Config + +In order to make this possible OwlDQ requires 2 drivers, MongoDB driver and UnityJDBC Driver. Out of the box OwlDQ comes preconfigured with these drivers. You simply open the MongoDB connection template and paste in your JDBC URL. + +``` +driverClass: mongodb.jdbc.MongoDriver + +path: /opt/owl/drivers/mongodb/ + +-- mongoJdbc.jar + +-- unityJDBC.jar +``` + +### Simply paste in JDBC Info + +![](<../../.gitbook/assets/Screen Shot 2020-08-01 at 10.09.20 AM.png>) + +### Discover Correlations, Relationships, DQ issues and Much More... + +![](<../../.gitbook/assets/Screen Shot 2020-08-01 at 10.10.45 AM.png>) + +The following table presents the various SQL statements related to table-level actions and the corresponding MongoDB statements.[https://docs.mongodb.com/manual/reference/sql-comparison/](https://docs.mongodb.com/manual/reference/sql-comparison/) + +![](<../../.gitbook/assets/Screen Shot 2020-08-02 at 2.17.34 PM.png>) + +### Limiting Collections in the JDBC URL + +``` +jdbc:mongodb://:@datalake0-dza1q.a.query.mongodb.net/?ssl=true&authSource=admin&rebuildschema=true&tables=orders +``` + +There are 3 collections in this mongodb atlas lake. By adding \&tables=orders in the URL params you can see only order collections show up in the explorer. + +![](<../../.gitbook/assets/Screen Shot 2021-07-22 at 1.54.04 PM.png>) + +### 3 Collections in MongoDB Atlas + +The total number of collections in mongodb atlas lake. + +![](<../../.gitbook/assets/Screen Shot 2021-07-22 at 1.57.29 PM.png>) diff --git a/dq-job-examples/owlcheck-spark/owlcheck-s3.md b/dq-job-examples/owlcheck-spark/owlcheck-s3.md new file mode 100644 index 00000000..da5af401 --- /dev/null +++ b/dq-job-examples/owlcheck-spark/owlcheck-s3.md @@ -0,0 +1,86 @@ +# DQ Job S3 + +S3 permissions need to be setup appropriately. + +{% hint style="info" %} +S3 connections should be defined using the root bucket. Nested S3 connections are not supported. +{% endhint %} + +#### Example Minimum Permissions + +``` +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "s3:ListBucketMultipartUploads", + "s3:ListBucket", + "s3:ListMultipartUploadParts", + "s3:PutObject", + "s3:GetObject", + "s3:GetBucketLocation" + ], + "Resource": [ + "arn:aws:athena:*::workgroup/primary", + "arn:aws:s3:::/*", + "arn:aws:s3:::", + "arn:aws:glue:*::catalog", + "arn:aws:glue:*::database/", + "arn:aws:glue:*::table//*" + ] + } + ] +} +``` + +(Needs appropriate driver) [http://central.maven.org/maven2/org/apache/hadoop/hadoop-aws/](http://central.maven.org/maven2/org/apache/hadoop/hadoop-aws/) Hadoop AWS Driver hadoop-aws-2.7.3.2.6.5.0-292.jar + +```bash +-f "s3a://s3-location/testfile.csv" \ +-d "," \ +-rd "2018-01-08" \ +-ds "salary_data_s3" \ +-deploymode client \ +-lib /home/ec2-user/owl/drivers/aws/ +``` + +### Databricks Utils Or Spark Conf + +```bash +val AccessKey = "xxx" +val SecretKey = "xxxyyyzzz" +//val EncodedSecretKey = SecretKey.replace("/", "%2F") +val AwsBucketName = "s3-location" +val MountName = "kirk" + +dbutils.fs.unmount(s"/mnt/$MountName") + +dbutils.fs.mount(s"s3a://${AccessKey}:${SecretKey}@${AwsBucketName}", s"/mnt/$MountName") +//display(dbutils.fs.ls(s"/mnt/$MountName")) + +//sse-s3 example +dbutils.fs.mount(s"s3a://$AccessKey:$SecretKey@$AwsBucketName", s"/mnt/$MountName", "sse-s3") +``` + +### Databricks Notebooks using S3 buckets + +```bash +val AccessKey = "ABCDED" +val SecretKey = "aaasdfwerwerasdfB" +val EncodedSecretKey = SecretKey.replace("/", "%2F") +val AwsBucketName = "s3-location" +val MountName = "abc" + +// bug if you don't unmount first +dbutils.fs.unmount(s"/mnt/$MountName") + +// mount the s3 bucket +dbutils.fs.mount(s"s3a://${AccessKey}:${EncodedSecretKey}@${AwsBucketName}", s"/mnt/$MountName") +display(dbutils.fs.ls(s"/mnt/$MountName")) + +// read the dataframe +val df = spark.read.text(s"/mnt/$MountName/atm_customer/atm_customer_2019_01_28.csv") +``` diff --git a/dq-job-examples/owlcheck-spark/owlcheck-snowflake.md b/dq-job-examples/owlcheck-spark/owlcheck-snowflake.md new file mode 100644 index 00000000..a903960b --- /dev/null +++ b/dq-job-examples/owlcheck-spark/owlcheck-snowflake.md @@ -0,0 +1,24 @@ +# DQ Job Snowflake + +## Example CMD Line + +``` +-h :5432/postgres \ +-drivermemory 4g \ +-master spark://:7077 \ +-ds PUBLIC.TRANSLATION \ +-deploymode client \ +-q "select * from PUBLIC.TRANSLATION" \ +-rd "2021-07-24" \ +-driver "net.snowflake.client.jdbc.SnowflakeDriver" \ +-cxn snowflake +``` + +### Example JDBC Connection URL + +jdbc:snowflake://\.snowflakecomputing.com?db=DEMODB\&warehouse=COMPUTE\_WH\&schema=PUBLIC + +### Drive Name + +net.snowflake.client.jdbc.SnowflakeDriver + diff --git a/dq-job-examples/owlcheck/README.md b/dq-job-examples/owlcheck/README.md new file mode 100644 index 00000000..db8dec81 --- /dev/null +++ b/dq-job-examples/owlcheck/README.md @@ -0,0 +1,217 @@ +# Advanced + +An OwlCheck is bash script that is essentially the launch point for any owl job to scan a dataset. A dataset can be a flat file (such as textfile, json file, parquet file, etc), or a table from any number of Databases (such as Oracle, Postgres, Mysql, Greenplum, DB2, SQLServer, Teradata, etc). + +Example Run a Data Quality check on any file by setting the file path. + +```bash +./owlcheck -ds stock_trades -rd 2019-02-23 -f /path/to/file.csv -d , +``` + +Example output below. A hoot is a valid JSON response + +```bash +{ + "dataset": "stock_trades", + "runId": "2019-02-03", + "score": 100, + "behaviorScore": 0, + "rows": 477261, + "passFail": 1, + "peak": 1, + "dayOfWeek": "Sun", + "avgRows": 0, + "cols": 5, + "activeRules": 0, + "activeAlerts": 0, + "runTime": "00:00:23", + "dqItems": {}, + "datashapes": [], + "validateSrc": [], + "alerts": [], + "prettyPrint": true +} +``` + +## Monthly Data + +Sometimes you may want to run monthly profiles with aggregated data. In this case the scheduling tool can supply the ${rd} as variable such as $runDate and the end date as $endDate. 1 line examples for bash or shell below. + +```bash +echo "Hello World Owl" + +runDate=$(date +"%Y-%m-%d") +endDate=$(date -d "$runDate +1 month" +%Y-%m-%d) + +echo $runDate +echo $endDate + +./owlcheck \ +-q "select * from table where date >= '$runDate' and date < '$endDate' " \ +-ds example \ +-rd $runDate \ +-tbin MONTH +``` + +## Monthly BackRun (Using Owl's built in Monthly) + +Owl has 2 convenient features here: 1) the use of built in ${rd} and ${rdEnd} removes the need for any shell scripting. 2) using -br, Owl will replay 20 months of data using this template automatically. + +```bash +./owlcheck \ +-q "select * from table where date >= '${rd}' and date < '${rdEnd}' " \ +-ds example +-rd 2019-01-01 +-rdEnd 2019-02-01 +-tbin MONTH +-br 20 +``` + +## Daily Data + +One of the most common examples is data loading or running once a day. A job control framework can pass in this value or you can pull it from shell. + +```bash +echo "Hello World Owl" + +runDate=$(date +"%Y-%m-%d") +echo $runDate + +./owlcheck \ +-q "select * from table where date = '$runDate' " \ +-ds example \ +-rd $runDate \ +-tbin DAY +``` + +## Daily Data (Using Owl's built in Daily) + +```bash +./owlcheck \ +-q "select * from table where date = '${rd}' " \ +-ds example \ +-rd 2019-03-14 +``` + +## Daily Data with Timestamp instead of Date + +```bash +./owlcheck \ +-q "select * from table where TS >= '${rd} 00:00:00' and TS <= '${rd} 23:59:59' " \ +-ds example \ +-rd 2019-03-14 +``` + +## OR Timestamp using ${rdEnd} + +```bash +./owlcheck \ +-q "select * from table where TS >= '${rd} 00:00:00' and TS < '${rdEnd} 00:00:00' " \ +-ds example \ +-rd 2019-03-14 \ +-rdEnd 2019-03-15 \ +-tbin DAY +``` + +## Hourly Data + +```bash +./owlcheck \ +-q "select * from table where TS >= '${rd}' and TS < '${rdEnd}' " \ +-ds example \ +-rd "2019-03-14 09:00:00" \ +-rdEnd "2019-03-14 10:00:00" \ +-tbin HOUR +``` + +## OwlCheck Template with Service Hook + +The best practice is to make a generic job that would be repeatable for every OwlCheck. Below is an example that first hits Owl using a REST call and then runs the response. + +```bash +curl -X GET "http://$host/v2/getowlchecktemplate?dataset=lake.loan_customer" \ +-H "accept: application/json" +``` + +The above REST call returns the below OwlCheck. It is left up to the Job Control to replace the ${rd} with the date from the Job Control system. You can use Owls built in scheduler to save these steps. + +```bash +./owlcheck \ +-lib "/home/danielrice/owl/drivers/mysql/" \ +-cxn mysql \ +-q "select * from lake.loan_customer where load_dt = '${rd}' " \ +-key post_cd_num -ds lake.loan_customer \ +-rd ${rd} \ +-dc load_dt -dl -dlkey usr_name,post_cd_num -dllb 5 \ +-tbin DAY -by DAY -dupe -dupeinc ip_address_home,usr_name -dupecutoff 85 \ +-fpgon -fpgkey usr_name,post_cd_num -fpgdc load_dt -fpglb 5 -fpgtbin DAY \ +-loglevel INFO \ +-h $host:5432/owltrunk \ +-owluser {user} +``` + +## REST API End Point + +The easiest option is to use the **runtemplate** end point API call to make requests to from cmdLine or JobControl System. This endpoint gets the OwlCheck saved in Owl instead of the client needing to know the OwlCheck details. + +{% swagger baseUrl="http://$host" path="/v2/runtemplate?dataset=lake.spotify" method="post" summary="RunTemplate" %} +{% swagger-description %} + +{% endswagger-description %} + +{% swagger-parameter in="path" name="dataset" type="string" %} +name of dataset. -ds OR opt.dataset +{% endswagger-parameter %} + +{% swagger-parameter in="path" name="rd" type="string" %} +yyyy-MM-dd format can add time or timezone. if note passed in it will use the current day +{% endswagger-parameter %} + +{% swagger-parameter in="path" name="rdEnd" type="string" %} +yyyy-MM-dd format can add time or timezone. if not passed it will not be used +{% endswagger-parameter %} + +{% swagger-response status="200" description="" %} +``` +{ + "msg": "Success, Owl Check is Running as process 13996", + "pid": "13996", + "runid": "2017-01-01", + "starttime": "Thu Oct 17 13:27:01 EDT 2019", + "cmd": "cmd": "-ds lake.spotify -rd 2019-10-17 -q \"select * from lake.spotify\" -cxn mysql -lib /opt/owl/drivers/mysql/ -drivermemory 2G -histoff -owluser {user}", + "dataset": "lake.spotify" +} +``` +{% endswagger-response %} +{% endswagger %} + +### Curl example for the above Rest Call + +```bash +TOKEN=$(curl -s -X POST http://$host/auth/signin -H "Content-Type:application/json" -d "{\"username\":\"$username\", \"password\":\"$password\"}" | jq -r '.token') + +curl -i -H 'Accept: application/json' \ + -H "Authorization: Bearer ${TOKEN}" \ + http://$host/v2/runtemplate?dataset=lake.spotify +``` + +### Bash Script + +A generic and repeatable owlcheck script for job schedulers, that hooks into Owl to get the template. + +```bash +#1 authenticate +curl -sb -X POST -d username={user} -d password={password} http://$OWL_HOST/login -c cookies.txt + +#2 get template +owlcheck_args=$(curl -b cookies.txt -H "accept: application/json" -X GET http://$OWL_HOST/v2/getowlcheckcmdlinebydataset\?dataset=insurance | sed 's/.*\[\(.*\)\]/\1/' | sed -e "s/^\"//" -e "s/\"$//" | sed 's/\\\"\(.*\)\\\"/\x27\1\x27/') + +#3 replace ${rd} with job_run_date +job_run_date="2019-03-14 10:00:00" +owlcheck_args=${owlcheck_args//'${rd}'/$job_run_date} + +#4 run owlcheck +eval owlcheck $owlcheck_args +``` + +For more Information on Owl's Scheduler check out the doc on **OwlCheck Cron** Page**.** diff --git a/dq-job-examples/owlcheck/add-date-column.md b/dq-job-examples/owlcheck/add-date-column.md new file mode 100644 index 00000000..494b2798 --- /dev/null +++ b/dq-job-examples/owlcheck/add-date-column.md @@ -0,0 +1,22 @@ +# Add Date Column + +## Example + +```bash +./owlcheck \ +-ds "datataset_date_column" \ +-rd "2019-07-01" \ +-f "/Users/Downloads/csv2/2019010.csv" \ +-adddc +``` + +{% hint style="info" %} + Add date column will use the run date supplied and add a date column named **OWL\_RUN\_ID** +{% endhint %} + +```bash +-adddc +``` + +_This is used when you are using datasets that do not contain a date column or a malformed date string_ + diff --git a/dq-job-examples/owlcheck/autoprofile.md b/dq-job-examples/owlcheck/autoprofile.md new file mode 100644 index 00000000..140c184e --- /dev/null +++ b/dq-job-examples/owlcheck/autoprofile.md @@ -0,0 +1,35 @@ +# AutoProfile + +**AutoProfile** allows you to select a set of databases and tables to quickly be cataloged. Each selected table will be profiled and added to the Owl Catalog via the selected agent. Various parameters like Alerts, Job Schedules, limits, and more can also be set. + +‌When you expand a datasource in the **Explorer** page, you're given a list of possible databases and their associated tables. **AutoProfile** is triggered when you select the ones you want and hit scan. This will take you to a separate page that allows you to configure the various AutoProfile parameters. + +A SparkSubmit will be launched for each table, so make sure the agent configuration is reasonable and the box has enough resources to handle each job. + +**Global Parameters** + +![Global Parameters](../../.gitbook/assets/global-params.png) + + **Histogram** and **Correlation**: Enable or Disable + + **PushDown**: Set metrics that will be run against the entire table, ignoring limit values. + + **Default Limit**: The default row limit to set for each table to be scanned. + + **Batch Size**: The number of concurrent SparkSubmit jobs the agent will be allowed to run. + + **Scan and Schedule** : Enable scheduling. + + **Alert**: Enable email alerts. + + **Agent**: The agent under which jobs will run. + +**Per Table Parameters** + +![](../../.gitbook/assets/table-params.png) + + **Columns**: The columns to select from (if none are specified, we assume all columns to be selected). + + **Date Filter**: The date column to be used as the `runDate` parameter when the job runs. + + **Default Limit**: Per table limits. diff --git a/dq-job-examples/owlcheck/cloudera-classpath.md b/dq-job-examples/owlcheck/cloudera-classpath.md new file mode 100644 index 00000000..6a82de53 --- /dev/null +++ b/dq-job-examples/owlcheck/cloudera-classpath.md @@ -0,0 +1,201 @@ +# Cloudera CLASSPATH + +## **What is a CLASSPATH?** + +A CLASSPATH is essentially a list of jars that get injected into a JVM on the start of a job execution. Like many applications, Spark can have jars injected when a job is run. Cloudera has defined a list of predefined jars (rightfully called classpath.txt): + +``` +/etc/spark2/conf/classpath.txt +``` + +that will get injected whenever Spark is called. Here is an example list of jars as defined within a cluster we have stood up @ Owl: + +``` +[danielrice@cdh-edge ~]$ cat /etc/spark2/conf/classpath.txt +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/activation-1.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/aopalliance-1.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/apacheds-i18n-2.0.0-M15.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/apacheds-kerberos-codec-2.0.0-M15.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/api-asn1-api-1.0.0-M20.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/api-util-1.0.0-M20.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/asm-3.2.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/avro-1.7.6-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/aws-java-sdk-bundle-1.11.134.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/azure-data-lake-store-sdk-2.2.9.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-beanutils-1.9.2.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-beanutils-core-1.8.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-codec-1.4.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-codec-1.9.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-configuration-1.6.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-daemon-1.0.13.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-digester-1.8.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-el-1.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-logging-1.2.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-math-2.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-math3-3.1.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/commons-net-3.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/core-3.1.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/curator-client-2.7.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/curator-framework-2.7.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/curator-recipes-2.7.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/disruptor-3.3.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/findbugs-annotations-1.3.9-1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/guava-11.0.2.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/guava-12.0.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/guice-3.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-annotations-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-ant-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-archive-logs-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-archives-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-auth-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-aws-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-azure-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-azure-datalake-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-common-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-datajoin-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-distcp-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-extras-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-gridmix-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-hdfs-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-hdfs-nfs-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-mapreduce-client-app-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-mapreduce-client-common-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-mapreduce-client-core-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-mapreduce-client-hs-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-mapreduce-client-nativetask-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-mapreduce-examples-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-nfs-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-openstack-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-rumen-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-sls-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-streaming-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-yarn-api-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-yarn-applications-distributedshell-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-yarn-client-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-yarn-common-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-yarn-registry-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-yarn-server-common-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-yarn-server-nodemanager-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hadoop-yarn-server-web-proxy-2.6.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hamcrest-core-1.3.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-annotations-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-client-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-common-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-examples-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-external-blockcache-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-hadoop-compat-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-hadoop2-compat-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-it-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-prefix-tree-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-procedure-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-protocol-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-resource-bundle-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-rest-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-rsgroup-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-server-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-shell-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hbase-thrift-1.2.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/high-scale-lib-1.1.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hsqldb-1.8.0.10.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/htrace-core-3.2.0-incubating.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/htrace-core4-4.0.1-incubating.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/httpclient-4.2.5.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/httpcore-4.2.5.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/hue-plugins-3.9.0-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jackson-annotations-2.2.3.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jackson-core-2.2.3.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jackson-core-asl-1.8.10.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jackson-databind-2.2.3-cloudera.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jackson-jaxrs-1.8.10.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jackson-mapper-asl-1.8.10-cloudera.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jackson-xc-1.8.10.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jamon-runtime-2.4.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jasper-compiler-5.5.23.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jasper-runtime-5.5.23.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/java-xmlbuilder-0.4.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/javax.inject-1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jaxb-api-2.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jaxb-api-2.2.2.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jaxb-impl-2.2.3-1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jcodings-1.0.8.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jets3t-0.9.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jettison-1.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jettison-1.3.3.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jline-2.11.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/joni-2.1.2.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jruby-cloudera-1.0.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jsch-0.1.42.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jsp-2.1-6.1.14.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jsp-api-2.1-6.1.14.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jsp-api-2.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/jsr305-3.0.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/leveldbjni-all-1.8.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/log4j-1.2.16.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/log4j-1.2.17.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/metrics-core-2.2.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/metrics-core-3.0.2.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/microsoft-windowsazure-storage-sdk-0.6.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/mockito-all-1.8.5.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/netty-3.10.5.Final.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/netty-all-4.0.50.Final.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/okhttp-2.4.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/okio-1.4.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/paranamer-2.3.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/protobuf-java-2.5.0.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/slf4j-api-1.7.5.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/slf4j-log4j12-1.7.5.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/snappy-java-1.0.4.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/spark-1.6.0-cdh5.16.1-yarn-shuffle.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/spymemcached-2.11.6.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/stax-api-1.0-2.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/xercesImpl-2.9.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/xml-apis-1.3.04.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/xmlenc-0.52.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/jars/zookeeper-3.4.5-cdh5.16.1.jar +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/LICENSE.txt +/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/NOTICE.txt +/usr/java/jdk1.8.0_131/lib/tools.jar +``` + +At each execution of any Spark job (including the use of spark-submit) this list of jars above will automatically get loaded. + +## What is a JAR? + +A jar file is essential a compressed list of classes and methods. It is important to note that when jar files are built they will typically have an associated version number. + +Someone can look at the contents of a jar file by executing + +``` +jar -tvf phoenix-4.13.1-HBase-1.3-client.jar +``` + +Or you can wrap the above in a for loop that will look at the contents of every jar that might contain a method you are looking for. + +``` +for i in `ls -1 *.jar`;do jar -tvf $i | grep -i htrace/trace;echo $i;done; +``` + +## Common issues that can occur with CLASSPATH's + +{% hint style="info" %} +**Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace**\ +at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)\ +at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)\ +at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)\ +at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)\ +at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)\ +at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.\(ConnectionManager.java:635) +{% endhint %} + +When a situation like this occurs it means that a method cannot be found in the classpath for the job that is trying to execute. This can indicate a couple things: + +1. The job cannot find a jar file that contains the method flagged (in the example above the org/apache/htrace/Trace method) +2. Sometimes different versions of the same jar file gets loaded and the first jar loaded will always win. Older jars that get loaded first may not have a method defined in new jars. + +At owl we have solved CLASSPATH / CLASSLOAD issues by automatically injecting jars defined in our _**owl/libs**_ directory, and allowing users the ability to simply toggle loading them or not. diff --git a/dq-job-examples/owlcheck/cluster-health-report.md b/dq-job-examples/owlcheck/cluster-health-report.md new file mode 100644 index 00000000..88274faa --- /dev/null +++ b/dq-job-examples/owlcheck/cluster-health-report.md @@ -0,0 +1,54 @@ +--- +description: How much is your redundant data costing you? +--- + +# Column Matching + +### Reclaim Gigabytes of Redundant Data + +As data engineers, first we copy files into a landing zone, next we load the files into a staging area. After that we transform (ETL) the data into the final table. Soon that same data is copied to a lake for other groups to run analytics on. Eventually a group of analysts will need the data in another format and a data engineer will copy the data in a newly joined or transposed fashion. Sounds familiar? + +The result is the same data or similar columns of the same data being copied many times. **The answer**: _Buy more hardware_... could be OR _run an Owl health report_ and gain an understanding of how much data could be removed, reclaiming disk space and instantly seeing a return on investment after clicking the button. + +### Tabular breakdown of % fingerprint match + +![](../../.gitbook/assets/owl-cluster-health.png) + +Sometimes its not as simple as comparing 2 tables from the same database. Owl allows a technical user to setup multiple DB connections before executing an owl health check. + +``` +import com.owl.common.Props +import com.owl.core.Owl + +val c1 = new Connection() +c1.dataset = "silo.account" +c1.user = "user" +c1.password = "pass" +c1.query = "select id, networth, acc_name, acc_branch from silo.account limit 200000" +c1.url = "jdbc:mysql://owldatalake.chzid9w0hpyi.us-east-1.rds.amazonaws.com:3306" + +val c2 = new Connection() +c2.dataset = "silo.user_account" +c2.user = "user" +c2.password = "pass" +c2.query = "SELECT acc_name, acc_branch, networth FROM silo.account limit 200000" +c2.url = "jdbc:mysql://owldatalake.chzid9w0hpyi.us-east-1.rds.amazonaws.com:3306" + +val props = new Props() +props.dataset = "colMatchTest1" +props.runId = "2017-02-04" +props.connectionList = List(c1,c2).asJava +props.colMatchBatchSize = 2 +props.colMatchDurationMins = 3 + +val matchDF = new Owl(props).colMatchDF +matchDF.show + +matchDF.createOrReplaceTempView("matches") +``` + +### High level view of data overlap + +![](../../.gitbook/assets/owl-health-chart.png) + + diff --git a/dq-job-examples/owlcheck/date-time-variable-options.md b/dq-job-examples/owlcheck/date-time-variable-options.md new file mode 100644 index 00000000..bd414cf8 --- /dev/null +++ b/dq-job-examples/owlcheck/date-time-variable-options.md @@ -0,0 +1,20 @@ +# Date Time Variable Options + +**Benefit**: Enhanced query and file date templating and variable options. This allows easier scheduling and programmatic templating for common date variables. + +| Key | Function | +| --------- | ---------------------------------------------------------------------------------------- | +| ${rd} |

replaces with -rd values in CMD, e.g. 2021-09-01

| +| ${rdEnd} | replaces with -rdEnd values in CMD | +| ${yyyy} | replaces with 4 digit year portion of -rd values in CMD, e.g. 2021 | +| ${yy} | replaces with 2 digit year portion of -rd values in CMD, e.g. 21 | +| ${M} | replaces with 1 digit month portion of -rd values in CMD | +| ${MM} | replaces with 2 digit month portion of -rd values in CMD | +| ${MMM} | replaces with 3 letter month name portion of -rd values in CMD, e.g. Jan, Jul, Dec | +| ${MMMMM} | replaces with long month name portion of -rd values in CMD, e.g. January, July, December | +| ${d} | replaces with 1 digit day portion of -rd values in CMD | +| ${dd} | replaces with 2 digit day portion of -rd values in CMD | +| ${HH} | replaces with 2 digit hour portion of -rd values in CMD | +| ${KK} | replaces with 1 digit hour portion of -rd values in CMD | +| ${mm} | replaces with 2 digit minute portion of -rd values in CMD | +| ${ss} | replaces with 2 digit second portion of -rd values in CMD | diff --git a/dq-job-examples/owlcheck/deploy-mode.md b/dq-job-examples/owlcheck/deploy-mode.md new file mode 100644 index 00000000..1fb64bf9 --- /dev/null +++ b/dq-job-examples/owlcheck/deploy-mode.md @@ -0,0 +1,30 @@ +--- +description: Yarn and Cluster +--- + +# Deploy Mode + +### Deploy Mode Client + +``` +--deploy-mode client +``` + +### Deploy Mode Cluster + +``` +--deploy-mode cluster +``` + +### Job Stuck in ACCEPTED State + +yarn.Client: Application report for application\_1557720962505\_0085 (state: ACCEPTED)\ +yarn.Client: Application report for application\_1557720962505\_0085 (state: ACCEPTED) + +If running in cluster mode make sure you are passing in the below + +```bash +--deploy-mode cluster +--master yarn # or spark master +-h 123.45.6.77:2181 # host to owl metastore +``` diff --git a/dq-job-examples/owlcheck/explorer.md b/dq-job-examples/owlcheck/explorer.md new file mode 100644 index 00000000..da8a3123 --- /dev/null +++ b/dq-job-examples/owlcheck/explorer.md @@ -0,0 +1,27 @@ +# Explorer (advanced) + +## Explore Database Connections and File systems + +Use the explorer tab to quickly see which tables are cataloged with Owl (the square catalog icon) and which have Owl's quality protection (the owl icon). + +Below you will see 48/48 database tables have been cataloged with Owl but only 21/48 have an owlcheck. This means that this particular database schema is 44% protected from future DQ issues. + +![](<../../.gitbook/assets/owl-explorer (1).png>) + +## DQ coverage over time + +As you add more datasets to Owl, you will see your bar chart increase over time and the donut chart fill in with more coverage. + +## Job Estimator + +Before firing off a large ML job it can be helpful to understand the amount of cores and ram that the job requires to run efficiently. Right sizing jobs is often the best way to get the best performance out of each run. Click the \[Auto Estimate] button for quick stats for both sizing and estimated runtime information. + +### Dynamic allocation + +Many clusters offer the ability to scale up and down job containers. If Dynamic Allocation is turned on you may not need or desire Owl's recommended `num-executors` or `executor-memory`. However, in our testing right sizing the job before executing is both faster and a healthy habit. Faster, because there is less orchestration and context switching while the job is doing work; we've minimized time spent running out of space and having to reallocate and shuffle to a new container. Healthier, because it give the user real-time feedback on the cost of each feature and the ability to control the cost benefit analysis. + +## SQL Editor + +Automatically tracks to the connection, database and table in the explorer and provides a quick way to ask the database simple questions like, counts, groupings and specific clauses. + +![](../../.gitbook/assets/owl-sql-editor.png) diff --git a/dq-job-examples/owlcheck/file-lookback.md b/dq-job-examples/owlcheck/file-lookback.md new file mode 100644 index 00000000..227591f3 --- /dev/null +++ b/dq-job-examples/owlcheck/file-lookback.md @@ -0,0 +1,40 @@ +# File Look Back + +As of 2021.11 this option is exposed in the Explorer under the DQ Job section. Users can persist (save) this option by clicking the Union Lookback checkbox. + +![Click the checkbox to enable -fllb flag](broken-reference) + +{% hint style="info" %} +File look back (-fllb) should only be used when a SQL layer is not available. This is considered for advanced use cases, but may not be suitable for all file types and folder structures. Best practice is to expose a date signature somewhere in the file or directory naming convention. +{% endhint %} + +## Example + +``` +-ds "demo_lookback" \ +-rd "2017-07-29" \ +-lib "/opt/owl/drivers/mysql" \ +-cxn "mysql" \ +-q "select * from lake.dateseries where DATE_COL = '2017-07-29' " \ +-dc DATE_COL \ +-dl \ +-dlkey sym \ +-dllb 4 \ +-fllb +``` + +{% hint style="info" %} +This look back will load your past 4 runs as your historical training set +{% endhint %} + +File look back is used with [deep learning ](../../dq-visuals/more/outliers.md#numerical-outliers)or [pattern matching](../../dq-visuals/more/pattern-mining.md). In the example above it is used with deep learning. + +``` +-fllb +``` + +_**This is often used with files and in conjunction with**_ [_**-adddc**_ ](add-date-column.md)_**in cases where a date column is not in an ideal format or you do not have a date column on the given dataset**_ + +_Despite the name, this can be used with file or database storage formats._ + +__ diff --git a/dq-job-examples/owlcheck/filter.md b/dq-job-examples/owlcheck/filter.md new file mode 100644 index 00000000..0e4ab6f8 --- /dev/null +++ b/dq-job-examples/owlcheck/filter.md @@ -0,0 +1,35 @@ +--- +description: Similar to a grep for limiting a dataframe to rows containing a substring +--- + +# Filter & Filter Not + +{% hint style="info" %} +This feature should only be used when -q (query) and -fq (filequery) are not applicable. This feature is primarily used with raw files for limited filtering and not advanced conditional logic. +{% endhint %} + +## Example + +```bash +./owlcheck \ +-ds "dataset_name" \ +-rd "2018-07-23 10" \ +-d "," \ +-f "/Users/Documents/file.csv" \ +-filter "2018-07-23" +``` + +{% hint style="info" %} +**-filter "2018-07-23"** + +If file.csv contained multiple strings, but you only wanted rows containing "2018-07-23" +{% endhint %} + +## The inverse + +{% hint style="info" %} +**-filternot "2018-07-23"** + +To exclude rows containing "2018-07-23" +{% endhint %} + diff --git a/dq-job-examples/owlcheck/job-kafka.md b/dq-job-examples/owlcheck/job-kafka.md new file mode 100644 index 00000000..b5d5332b --- /dev/null +++ b/dq-job-examples/owlcheck/job-kafka.md @@ -0,0 +1,99 @@ +# DQ Job Kafka + +### Kafka Requires Zookeeper + +Apache Kafka typically requires zookeeper. This file and cmd can be run from inside /kafka/bin + +```python +# Start the ZooKeeper service +# Note: Soon, ZooKeeper will no longer be required by Apache Kafka. +$ bin/zookeeper-server-start.sh config/zookeeper.properties +``` + +### Start a Kafka Server + +Precursor step to Owl (you likely already have this step completed if you use Kafka) + +```scala +bin/kafka-server-start.sh config/server.properties +``` + +### Start a Kafka Topic + +Precursor step to Owl (you likely already have this step completed if you use Kafka) + +```python +bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test + +# prefered cmd is below +bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test +``` + +### Put a msg on "test" Topic + +```scala +bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test +``` + +### Kafka Consumer or Owl Consumer + +Kafka works as a topic so you can have many consumers. Here is a basic cmdline consumer but we can add Owl as a second consumer. + +```scala +bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning +``` + +```scala +/opt/owl/bin/owlcheck.sh + -kafkatopic test + -ds machine1 + -streamformat csv + -kafkaport 9092 + -kafkabroker localhost + -streaminterval 60 + -stream -kafka + -header first_name + -master local +``` + +![](../../.gitbook/assets/owl-sensor-streams.png) + +![](<../../.gitbook/assets/Screen Shot 2021-07-06 at 9.33.58 AM.png>) + +### Streams vs Sensors + +Technically speaking anything moving in real-time is a stream of data but Owl classifies streams and IoT sensors as slightly different for the following reasons: + +#### Sensors + +Sensors are commonly a standard time-series. Signal, Time, Value + +| Signal | Time | Value | +| ----------- | ------------------- | ----- | +| device1-CPU | 2019-02-11 13:40:55 | 4 | +| device1-CPU | 2019-02-11 14:33:20 | 2 | + +#### Streams + +Streams commonly look like messages, jsons, avro or batch data but constantly flowing. Another way to think of it is a multiple time-series + +``` +[ + trade: { + price: 23.75, + qty: 20, + symbol: HDP + }, + trade: { + } +] +``` + +| fname | age | networth | email | +| ----- | --- | -------- | -------------- | +| Joe | 45 | $130,000 | joe@yahoo.com | +| Mark | 33 | $125,000 | mark@yahoo.com | + +The difference between a Sensor and a Stream in the above example is that in the case of the sensor the user is primarily concerned with the actual value of the "Value". Meaning a spike in temperature or a drop in CPUs. But in a stream of customer data there isn't a time "X" and value "Y" there are many values "Y" and you a user is interested in the overall quality of both the entire stream and the individual values. Relationship analysis and other correlative functions apply here. If you were to chart a "stream" what would you chart? The row count volume or just one of the columns or the count of something? But if you were to chart a sensor you know exactly what you would chart... the "Value" over "Time". + +Fortunately Owl has already thought and worked through the many nuances required to understand, monitor and predict accurately for all of these use-case. All that is required is to subscribe the stream. diff --git a/dq-job-examples/owlcheck/multiple-pattern-combinations.md b/dq-job-examples/owlcheck/multiple-pattern-combinations.md new file mode 100644 index 00000000..6b1fc544 --- /dev/null +++ b/dq-job-examples/owlcheck/multiple-pattern-combinations.md @@ -0,0 +1,36 @@ +--- +description: Run more than one relationship through pattern matching +--- + +# Multiple Pattern Relationships + +## Example + +```bash +./owlcheck \ +-ds "fpg_multiple" \ +-rd "2018-10-04" \ + +-cxn "postgres" \ +-lib "/opt/owl/drivers/postgres/" \ +-q "select * from public.fpg_accounts where d_date = '2018-10-04'" \ + +-fpgon \ +-fpgdc "d_date" \ +-fpglb "4" \ +-fpgmulti "id,ssn_num=first_name,email|id,ssn_num=first_name,gender|id,ssn_num=last_name" +``` + +{% hint style="info" %} +Instead of -fpgkey and -fpgcol + +key1=cols1|keys2=cols2 + +Enter multiple key=cols combinations separated by a pipe +{% endhint %} + +```bash +-fpgmulti "id,ssn_num=first_name,email|id,ssn_num=first_name,gender" + +``` + diff --git a/dq-job-examples/owlcheck/owlcheck-43m-rows.md b/dq-job-examples/owlcheck/owlcheck-43m-rows.md new file mode 100644 index 00000000..62cf06e7 --- /dev/null +++ b/dq-job-examples/owlcheck/owlcheck-43m-rows.md @@ -0,0 +1,24 @@ +# DQ Job 43M rows + +Owl commonly benchmarks on large daily datasets. In this case a 43 million row table with 12 columns completes in under 6 mins (5:30). The best balance for this dataset was 3 executors each with 10G of ram. + +``` +./owlcheck \ +-u user -p password \ +-c jdbc:mysql://owldatalake.chzid9w0hpyi.us-east-1.rds.amazonaws.com:3306 \ +-q "select * from silo.account_large where acc_upd_ts > '2018-02-01 05:0:00'" \ +-rd 2019-02-02 \ +-ds account_large \ +-dc acc_upd_ts \ +-corroff \ +-histoff \ +-driver com.mysql.cj.jdbc.Driver \ +-lib "/home/ec2-user/owl/drivers/mysql/" \ +-master yarn \ +-deploymode client \ +-numexecutors 3 \ +-executormemory 10g \ +-histoff -corroff -loglevel DEBUG -readonly +``` + +_note: not all Owl features were turned on during this run. On large datasets it is worth it to consider limiting the columns, owl-features, or lookbacks if they are not of interest._ diff --git a/dq-job-examples/owlcheck/owlcheck-backrun.md b/dq-job-examples/owlcheck/owlcheck-backrun.md new file mode 100644 index 00000000..b72d8cdd --- /dev/null +++ b/dq-job-examples/owlcheck/owlcheck-backrun.md @@ -0,0 +1,53 @@ +--- +description: Replay 30 days of data +--- + +# DQ Job Back Run + +### How to Replay a Data Test + +Many times you will want to see how a dataset plays out over time. This could be 5 days or 5 months. Using this slider the tool will automatically create training sets and profiles as well as run any rules or outliers you've put in place. + +![](<../../.gitbook/assets/Screen Shot 2021-04-27 at 8.14.28 AM.png>) + +### Quickly Replay 30 days of data, -br 30 + +Add -br to any owlcheck and replay in time order. Jan 1, Jan 2, Jan 3... To do this we need to use the ${rd} variable that owl provides as a run\_date replacement for job control and templates. Also note that if you run from the cmdline you need to escape "$"s. so use \\${rd}. If you are running from a Notebook or Java or Scala or the Rest API you do not need to escape the ${rd} variable. + +```bash +./owlcheck \ +-ds OWLDB2.NYSE_STOCKS3 -rd "2018-01-14" \ +-lib "/opt/owl/drivers/db2/" \ +-cxn db2 \ +-q "select * from OWLDB2.NYSE_STOCKS where TRADE_DATE = '\${rd}'" \ +-br 4 +``` + +### Replay 4 Months of data, -br 4 -tbin MONTH + +In situations where your data rolls up into Months you may want to re-run several months of data but not a day at a time. In this case we will use -br with -tbin + +```bash +./owlcheck \ +-ds OWLDB2.NYSE_STOCKS3 \ +-rd "2018-01-01" \ +-q "select * from OWLDB2.NYSE_STOCKS where TRADE_DATE = '\${rd}'" \ +-br 4 \ +-tbin MONTH \ +-lib "/opt/owl/drivers/db2/" \ +-cxn db2 +``` + +### Monthly using a range for the entire Month + +```bash +./owlcheck \ +-ds OWLDB2.NYSE_STOCKS3 \ +-rd "2018-01-01" \ +-rdEnd "2018-02-01" \ +-q "select * from OWLDB2.NYSE_STOCKS where TRADE_DATE >= '${rd}' and TRADE_DATE < '${rdEnd}'" \ +-br 4 \ +-tbin MONTH +-lib "/opt/owl/drivers/db2/" \ +-cxn db2 +``` diff --git a/dq-job-examples/owlcheck/owlcheck-cron.md b/dq-job-examples/owlcheck/owlcheck-cron.md new file mode 100644 index 00000000..00038b89 --- /dev/null +++ b/dq-job-examples/owlcheck/owlcheck-cron.md @@ -0,0 +1,62 @@ +--- +description: Template for Job Control +--- + +# DQ Job Cron + +### Cron / Autosys / Control M / Oozie + +It is common for organization to need to run jobs on a schedule. Below are a few shell tricks to get a date from bash and use an OwlCheck with template variables. + +Kinit and get run\_date from shell or job control variable, pass it into Owl using $run\_date + +```bash +%sh +run_date=$(date +%Y-%m-%d) +run_date=$(date '+%Y-%m-%d %H:%M') +echo $run_date + +#kinit +echo "password" | kinit userabc@CW.COM + +~/owl/bin/owlcheck -q "select * from lake.stock_eod where date = '$run_date' " \ +-u user -p pass \ +-c "jdbc:mysql://owldatalake.chzid9w0hpyi.us-east-1.rds.amazonaws.com:3306" \ +-rd "$run_date" \ +-dc "date" \ +-dl \ +-dllb 7 \ +-dlminhist 2 \ +-tbin DAY \ +-dlkey sym,exch \ +-ds "lake.stock_nasdaq" \ +-driver "com.mysql.jdbc.Driver" \ +-lib "/home/ec2-user/owl/drivers/mysql/" \ +-master yarn -deploymode client -numexecutors 1 -executormemory 1g \ +-loglevel DEBUG +``` + +### Template + +You can also use -template to use Owl as a service hook and remove the need to pass in almost anything. In this case Owl will look up the template automatically from either a previous run or if you've saved a template, and use these variables. Any variable at the cmdline will override and win/replace. This is a great way to remove connection and other information from being hard coded into the job control framework and allows edit ability from Owl Webapp. + +```bash +%sh +~/owl/bin/owlcheck -usetemplate -ds lake.stock_nasdaq -rd $run_date +``` + +### Owl Scheduler - Built In + +A quick option is to use Owl's built in scheduler. Owl will automatically substitute the runtime variables like ${rd} into the job. This also gives you control to edit the OwlCheck. + +![](../../.gitbook/assets/owl-schedule.png) + +The schedule is based on the OwlCheck Template. This way the runtime variables are replaced in each run. Notice the ${rd} below. + +![](../../.gitbook/assets/owl-template-schedule.png) + +### All Scheduled Jobs in 1 Place + +Under the jobs dashboard you can see an overview schedule with all running jobs and their status. + +![](../../.gitbook/assets/owl-scheduler.png) diff --git a/dq-job-examples/owlcheck/owlcheck-linkid.md b/dq-job-examples/owlcheck/owlcheck-linkid.md new file mode 100644 index 00000000..5cc081c1 --- /dev/null +++ b/dq-job-examples/owlcheck/owlcheck-linkid.md @@ -0,0 +1,104 @@ +# DQ Job LinkId + +Ability to link an OwlCheck findings back to the source record for remediation. The linkId needs to be unique, commonly the primary key. There are 2 ways to provide the linkId, 1) a the cmdline via -linkid or 2) in a notebook via opt.linkId. Owl supports one or many primary key columns in your datasets for record linkage to your original table, file or dataframe. If your primary key column contains many columns use a comma to delineate. + +### Notebook + +```scala +val opt = new OwlOptions() +opt.runId = "2018-02-24" +opt.dataset = "orders" +opt.linkId = Array("transaction_id", "trans_time") +``` + +### CmdLine + +```bash +./owlcheck -ds orders \ +-rd "2018-02-24" \ +-linkid transaction_id,trans_time +``` + +{% hint style="info" %} +For rules to use linkid the columns need to be present in the select statement (either select \* or select specific column names). All Simple rules are eligible for linkid and Freeform rules need to contain the columns in the projection part of the SQL statement. +{% endhint %} + +### Activity Usage + +| **Activity** | **Supported** | **Description** | +| ------------ | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **SHAPE** | YES | 1 Example of each shape issue will have a link back to the corrupt record for remediation. | +| **OUTLIER** | YES | Each outlier will have a link back to the detected record for remediation. If you apply a limit you will only get the limited amount. Not on categorical. | +| **DUPE** | YES | Each duplicate or fuzzy match will have a link back to the record for remediation. | +| **SOURCE** | PARTIAL | Each source record that has a cell value that doesn't match to the target will have a link for remediation. SOURCE will not have links for row counts and schema as these are not record level findings. | +| **RULE** | YES | Break records for Freeform and Simple rule types will be stored (any records that did not meet the condition of the RULE will be provided with the linkId columns). These are stored as delimited strings in the rule\_breaks table along with the dataset, run\_id and rule name. Please note when using Freeform SQL the linkId columns should be part of the select statement. LinkId columns should be unique identifiers. | +| **BEHAVIOR** | NO | This class of data change is when a a section of your data is drifting from its normal tendency there is no 1 record to link. | +| **SCHEMA** | NO | This class of data change is at a schema/dataset level there are no records to link. | +| **RECORD** | PARTIAL | In some cases when a record is added or removed it may be available for linking. | +| **PATTERN** | NO | Patterns are not always a direct link. This item is still under performance review. | + +### Notebook API Example + +``` ++------------+----------+-------+-------+-----+-----------------+---------------+ +| dataset| runId|fieldNm| format|count| percent| transaction_id| ++------------+----------+-------+-------+-----+-----------------+---------------+ +| order |2018-02-24| fname|xxxx'x.| 1|7.142857142857142|t-1232 | ++------------+----------+-------+-------+-----+-----------------+---------------+ +``` + +```scala +owl.getShapesDF +``` + +### Rest API Example + +When supplying a linkId Owl will naturally exclude this field from most activities. Meaning a unique ID or primary key column can not be duplicative or it would not be the primary key, hence it will not be evaluated for duplicates. The same goes for Outliers and Shapes as a large sequence number or other variations might trigger a false positive when this column is denoted to be simply for the purpose of linking uniquely back to the source. If you for some reason want to also evaluate this column as well as link it please create a derived column with a different name and Owl will naturally handle both cases. + +```scala +owl.getShapes +owl.getDupes +owl.getOutliers +owl.getRuleBreaks +owl.getSourceBreaks +``` + +### getRules() + +```scala +----Rules---- ++-----------------+----------+--------------------+------------------+------+ +| dataset| runId| ruleNm| ruleValue|linkId| ++-----------------+----------+--------------------+------------------+------+ +|dataset_outlier_3|2018-02-24| fname_like_Kirk|fname like 'Kirk' | c-41| +|dataset_outlier_3|2018-02-24| fname_like_Kirk|fname like 'Kirk' | c-42| +|dataset_outlier_3|2018-02-24| fname_like_Kirk|fname like 'Kirk' | c-43| +|dataset_outlier_3|2018-02-24| fname_like_Kirk|fname like 'Kirk' | c-44| +|dataset_outlier_3|2018-02-24| fname_like_Kirk|fname like 'Kirk' | c-45| +|dataset_outlier_3|2018-02-24|if_email_is_valid...| email| c-31| +|dataset_outlier_3|2018-02-24|if_email_is_valid...| email| c-33| +|dataset_outlier_3|2018-02-24|if_zip_is_valid_Z...| zip| c-40| ++-----------------+----------+--------------------+------------------+------+ +``` + +### getDupes() + +first split on \~\~ then if you have a multiple part key split on \~| + +```scala +----Dupes---- ++-----------------+----------+-----+--------------------+----------+ +| dataset| runId|score| key| linkId| ++-----------------+----------+-----+--------------------+----------+ +|dataset_outlier_3|2018-02-24| 100|9ec828d5194fa397b...|c-45~~c-36| +|dataset_outlier_3|2018-02-24| 100|1f96274d1d10c9f77...|c-45~~c-35| +|dataset_outlier_3|2018-02-24| 100|051532044be286f99...|c-45~~c-44| +|dataset_outlier_3|2018-02-24| 100|af2e96921ae53674a...|c-45~~c-43| +|dataset_outlier_3|2018-02-24| 100|ad6f04bf98b38117a...|c-45~~c-42| +|dataset_outlier_3|2018-02-24| 100|1ff7d50a7a9d07d02...|c-45~~c-41| +|dataset_outlier_3|2018-02-24| 100|6ed858ed1f4178bb0...|c-45~~c-40| +|dataset_outlier_3|2018-02-24| 100|d2903703b348fb4cb...|c-45~~c-39| +|dataset_outlier_3|2018-02-24| 100|24bf54412de1e720d...|c-45~~c-38| +|dataset_outlier_3|2018-02-24| 100|7a7ce0beb41b39564...|c-45~~c-37| ++-----------------+----------+-----+--------------------+----------+ +``` diff --git a/dq-job-examples/owlcheck/owlcheck-transform.md b/dq-job-examples/owlcheck/owlcheck-transform.md new file mode 100644 index 00000000..46b8ba61 --- /dev/null +++ b/dq-job-examples/owlcheck/owlcheck-transform.md @@ -0,0 +1,37 @@ +# Transform + +### Transform Date + +During DQ setup you can transform columns such as Dates and Numbers to preferred formats. It is a common need to replace N.A. with nulls or empty white space. + +![](<../../.gitbook/assets/Screen Shot 2021-03-29 at 5.11.02 PM.png>) + + + +## Example + +``` +./owlcheck \ +-ds "dataset_transform" \ +-rd "2018-01-31" \ +-f "/Users/Documents/file.csv" \ +-transform "purch_amt=cast(purch_amt as double)|return_amt=cast(return_amt as double)" +``` + +{% hint style="info" %} +Submit an expression to transform a string to a particular type + +In this example, transform the purch\_amt column to a double +{% endhint %} + +```bash +-transform "purch_amt=cast(purch_amt as double)" +``` + +{% hint style="info" %} +Example of converting a string to a date +{% endhint %} + +```bash +-transform "RECEIVED_DATE=to_date(CAST(RECEIVED_DATE AS STRING), 'yyyyMMdd') as RECEIVED_DATE" +``` diff --git a/dq-job-examples/owlcheck/owlcheck-validate-source.md b/dq-job-examples/owlcheck/owlcheck-validate-source.md new file mode 100644 index 00000000..967fd692 --- /dev/null +++ b/dq-job-examples/owlcheck/owlcheck-validate-source.md @@ -0,0 +1,114 @@ +# DQ Job Validate Source + +### Reconciliation + +Commonly data driven organizations have a need to ensure that 2 tables or a table and file match. This match might be a daily reconciliation or any snapshot in time. Owl calls this Source to Target or Left to Right matching. It covers row differences, schema differences and all cell values. + +![](<../../.gitbook/assets/Screen Shot 2019-10-01 at 8.40.33 PM.png>) + +### Impala/Hive -> DB2 + +Below is an example of comparing a table in DB2 to the same table in Impala. + +```bash +./owlcheck \ +-lib "/home/install/owl/drivers/db2" \ +-cxn db2 \ +-q "select * from OWLDB2.NYSE_STOCKS where TRADE_DATE = '${rd}' " \ +-ds NYSE_STOCKS_VS \ +-rd "2018-01-10" \ +-vs \ +-valsrckey SYMBOL \ +-validatevalues \ +-h $host/owltrunk \ +-srcq "select * from nyse where TRADE_DATE = '${rd}' " \ +-srccxn impala-jdbcuser \ +-libsrc /home/isntall/owl/drivers/hivedrivers \ +-jdbcprinc jdbcuser@CW.COM -jdbckeytab /tmp/jdbcuser.keytab \ +-owluser admin \ +-executorcores 4 -numexecutors 6 -executormemory 4g -drivermemory 4g -master yarn -deploymode cluster \ +-sparkkeytab /home/install/owl/bin/user2.keytab \ +-sparkprinc user2@CW.COM +``` + +### DB2 -> Hive (Native) + +Most databases only expose data through a JDBC connection but Hive offers a second path which does not require a JDBC connection. Hive has the ability to push down its processing to the local worker nodes and read directly from disk in the case when the processing is happening locally on a cluster. If your processing is not happening local to the cluster then you must use HiveJDBC. Take note of the -hive flag. + +```bash +./owlcheck \ +-hive \ +-q "select * from nyse" \ +-ds hiveNativeNyse \ +-rd "2019-10-01" \ +-vs \ +-valsrckey exch,symbol,trade_date \ +-validatevalues \ +-srcq "select * from OWLDB2.NYSE_STOCKS" \ +-srccxn db2 -libsrc /home/install/owl/drivers/db2 \ +-numexecutors 2 -executormemory 5g -drivermemory 4g -master yarn -deploymode cluster \ +-sparkkeytab /home/install/owl/bin/user2.keytab -sparkprinc user2@CW.COM +``` + +### MySQL -> Oracle + +This example compares the entire table instead of just a single day. Notice the 3 part valsrckey EXCH,SYMBOL,TRADE\_DATE. Adding the date field ensures our key is unique and won't create a cartesian product. If the goal was to compare day over day with Oracle make sure to add TO\_DATE('YYYY-MM-DD', '2019-10-01') to the where clause. + +```bash +./owlcheck \ +-lib /home/install/owl/drivers/mysql/ \ +-cxn mysql \ +-q "select * from lake.nyse" \ +-ds lake.nyse \ +-rd 2019-10-01 \ +-vs \ +-valsrckey EXCH,SYMBOL,TRADE_DATE \ +-validatevalues \ +-sparkkeytab /home/install/owl/bin/user2.keytab \ +-sparkprinc user2@CW.COM \ +-srcq "select * from SYSTEM.NYSE" \ +-srccxn oracle \ +-libsrc /home/danielrice/owl/drivers/oracle/ +-numexecutors 2 -executormemory 5g -drivermemory 4g -master yarn -deploymode cluster \ +``` + +### File -> MySQL Table + +Taking a file and loading it into a staging table or final table is a common part of every ETL process. However it is extremely common that the file values do not match or coherence into the table properly and these silent errors are usually not caught until a business user sees the data far long down stream. + +```bash +./owlcheck \ +-ds lake.nyse \ +-rd 2019-10-01 \ +-cxn "mysql" \ +-q "select * from lake.nyse" \ +-vs \ +-valsrckey EXCH,SYMBOL,TRADE_DATE \ +-validatevalues \ +-srcfile "hdfs:///user/source/nyse.csv" \ +-srcd "," \ +-lib /home/install/owl/drivers/mysql/ \ +-sparkkeytab /home/install/owl/bin/user2.keytab \ +-sparkprinc user2@CW.COM \ +-numexecutors 2 -executormemory 5g -drivermemory 4g -master yarn -deploymode cluster \ +``` + +### File -> File + +Owl can compare a File to a File. This is common in landing zones and staging areas where a file might be moved or changed and you need to know if anything changed or is incorrect. + +```bash +./owlcheck \ +-ds lake.nyse \ +-rd 2019-10-01 \ +-f "hdfs:///user/target/nyse.csv" \ +-d "," \ +-vs \ +-valsrckey EXCH,SYMBOL,TRADE_DATE \ +-validatevalues \ +-srcfile "hdfs:///user/source/nyse.csv" \ +-srcd "," \ +-sparkkeytab /home/install/owl/bin/user2.keytab \ +-sparkprinc user2@CW.COM \ +-numexecutors 2 -executormemory 5g -drivermemory 4g -master yarn -deploymode cluster \ +``` diff --git a/dq-job-examples/owlcheck/zero-if-null.md b/dq-job-examples/owlcheck/zero-if-null.md new file mode 100644 index 00000000..b68798dc --- /dev/null +++ b/dq-job-examples/owlcheck/zero-if-null.md @@ -0,0 +1,29 @@ +# Nulls in Datasets + +## Example + +``` +./owlcheck \ +-ds "datataset_date_column" \ +-rd "2019-07-01" \ +-f "/Users/Downloads/csv2/2019010.csv" \ +-zfn \ +-nulls "N.A." +``` + +Zero if null will replace null values with zero + +``` +-zfn +``` + +To replace characters that represent a null to actual null values + +``` +-nulls "N.A." +``` + + + + + diff --git a/dq-visuals/behaviors.md b/dq-visuals/behaviors.md new file mode 100644 index 00000000..4ef63765 --- /dev/null +++ b/dq-visuals/behaviors.md @@ -0,0 +1,89 @@ +--- +description: >- + This is commonly referred to as statistical change detection or data + observability. +--- + +# Behavior (automatic) + +{% hint style="info" %} +Results are found under the Behavior tab (short for behavioral analytics). This will track changes in the heuristics of the underlying data profiling metrics. The adaptive rules modal will display a complete list of monitoring types and criteria. +{% endhint %} + +![](../.gitbook/assets/behaviors.gif) + +## Evolution of Rule based Data Quality + +The main goal of OwlDQ is to provide enterprise data quality insight while greatly reducing the volume of Rules that need to be written manually. When a dataset is brought under management, Owl profiles the data and builds a model for each dataset. This allows Owl to learn what "normal" means within the context of each dataset. As the data changes, the definition of "normal" will change as well. Instead of requiring the user to adjust rule settings, Owl continues to adjust its model. This approach enable Owl to provide automated, enterprise grade DQ coverage that removes the need to write dozens or even hundreds of rules per dataset. + +{% hint style="info" %} +Behaviors is turned on by default. Monitoring will calibrate and detect DQ observations, based on the profiling activity. +{% endhint %} + +![](../.gitbook/assets/adaptive\_rules.gif) + +## Using Behavioral Analytics (Change Detection) + +Typically, data quality checks are scheduled to run on a given dataset daily. Behavior DQ or simply put change detection, is built on top of data calculated by the Profile activity. The default settings will often work just fine, however, Owl allows the user to specify two key parameters: + +* Behavior Lookback - Number of Owlchecks that model will encompass. Lookback of 10 means that the model will be based on the combined statistics from the last 10 Owlchecks of this dataset. +* Learning Phase - Minimum number of Owlchecks before the Behavior model should begin to apply. Owl will not attempt to apply Behavioral scoring to a dataset until at least this many Owlchecks have been run on it. +* Applicable Behavioral Factors - The user can choose to forego the application of any of the above factors to the scoring of the model. For example, the user can instruct Owl to not track MIN and MAX ranges of values in columns by unchecking the MIN and MAX checkbox. This would prevent Owl from detecting any extreme values in any column of the dataset using the Behavioral model. + +![](<../.gitbook/assets/Screen Shot 2020-07-08 at 12.28.39 PM.png>) + +With each run, Owl will profile the dataset at the column level and begin to establish a model for the dataset. Initially, there is no need for any manual intervention, just keep the data coming. Within a few runs, the model will become sufficiently robust to begin detecting data quality issues that would otherwise be covered by manual rules. For example, Owl may detect that particular column experienced a spike in the number of NULL values (typical manually defined rule). + +{% hint style="info" %} + Owl's behavioral model consist of the following factors: + +* NULL values +* Empty values +* Cardinality +* Datatype shifting +* Row counts +* Load time +* Minimum Value +* Maximum Value +* Mean Value \[New Feature] +{% endhint %} + +Over time, the definition of normal for any given column within the dataset can change. The data may legitimately become more sparse or decrease in volume. Owl will continue to learn and adjust the model throughout the life of the dataset. However, if there is a drastic (but legitimate) change in the data, this could still mean several days of unnecessary alerts while the model is adjusting. To accelerate model adjustment, Owl provides the ability to adjust the acceptable range that for a given behavioral finding. + +For example, Owl learned that a particular column typically has between 10% and 20% Empty values. Today, the column is 80% Empty values. Owl raises a data quality issue and subtracts a proportional amount or points from the quality score of today's DQ run. The user may review the finding and realize that there is a legitimate business reason why that column has more empty values. With a few clicks, the user adjusts the acceptable range for that finding. Owl incorporates the user defined inputs into the model and adjust the current day's quality score. Owl would have eventually arrived at the correct range without any input, but without user input, it may have taken a few runs get there. + +![](<../.gitbook/assets/Screen Shot 2020-05-07 at 8.02.53 PM.png>) + +#### Drill-in to see the predicted range of valid values + +Automatic flagging of break records with erroneous data. + +![](<../.gitbook/assets/screen-shot-2021-04-27-at-8.07.58-am (1).png>) + +The screenshot above shows some of the controls and visualizations that can be used to tune the Behavioral model. In this specific example, Owl has detected that the cardinality of the EXCH field has doubled from 1 to 2 unique values. However, a user can instruct Owl to disregard this finding and adjust the model by manually specifying the range of values acceptable in this column. To assist the user, Owl provides a line chart and a historical topN visualization of this column's cardinality. + +![](<../.gitbook/assets/Screen Shot 2020-05-07 at 8.17.14 PM.png>) + +If the user wants to instruct Owl that there can be as many as 3 valid values in the EXCH column, the user click the "Manual" button and adjust the upper bound from 1 to 3, then click the save button. + +![](<../.gitbook/assets/Screen Shot 2020-05-07 at 8.17.43 PM.png>) + +Owl will adjust the Behavioral model's baseline, remove the the finding, and adjust the quality score. Going forward, Owl will know the acceptable range for unique values in the EXHC column is between 1 and 3. + +## Adaptive Rules + +As Owl builds and evolves the behavioral model, it will expose all of the "Adaptive Rules" that it is learning about. The above example demonstrates how Owl learns and automatically applies rules. The user has control but if left alone, Owl will learn what what "Normal" means for a given dataset and score quality of that dataset accordingly. This will result in a large set of rules that are automatically applied and adapted as the dataset changes over time. + +To view or modify Adaptive Rules, navigate to the Behavior tab on the "Hoot" page for the desired dataset and click the "View AR" button on the right side of the screen. This will bring up a full list of Adaptive Rules. + +![](<../.gitbook/assets/Screen Shot 2020-05-07 at 8.37.37 PM.png>) + +The Adaptive Rules also provides the user with the ability to adjust ranges derived from the behavior model. The user can manually adjust the tolerance range ad well as the score of any Adaptive Rule. While this may at times be convenient, it is also just fine to let Owl handle the model tuning through its own learning process. + +### Scoring + +In adaptive mode Owl automatically generates a DQ item score based on the egregiousness of the line item. This measurement is directly proportional to the distance from the green range to the red line. Example below. + +![](../.gitbook/assets/owl-behavior-score.png) + +The score can range from 0-30. This ties to the percent change and Z-Score. In cases when the Z-Score ranges from 0.0 - 6.0. diff --git a/dq-visuals/more/README.md b/dq-visuals/more/README.md new file mode 100644 index 00000000..5135edfd --- /dev/null +++ b/dq-visuals/more/README.md @@ -0,0 +1,7 @@ +--- +description: Collibra DQ advanced features +--- + +# More + +When specific DQ challenges require specific DQ detection techniques, Collibra DQ offers a wide variety of advanced functionality. While Schema and Shapes utilize auto-discovery, other detection algorithms are best suited for users that understand their data and have specific use-cases in mind. Read more to understand if specific dimensions can be applied to your data. diff --git a/dq-visuals/more/duplicates.md b/dq-visuals/more/duplicates.md new file mode 100644 index 00000000..5c14036e --- /dev/null +++ b/dq-visuals/more/duplicates.md @@ -0,0 +1,39 @@ +# Duplicates (advanced) + +{% hint style="info" %} +This is an advanced opt-in feature +{% endhint %} + +## General Ledger. Accounting use-case + +{% embed url="https://owl-analytics.com/general-ledger" %} + +Whether you're looking for a fuzzy matching percent or single client cleanup, Owl's duplicate detection can help you sort and rank the likelihood of duplicate data. + +![](../../.gitbook/assets/owl-dupe-booked.png) + +```bash +-f "file:///home/ec2-user/single_customer.csv" \ +-d "," \ +-ds customers \ +-rd 2018-01-08 \ +-dupe \ +-dupenocase \ +-depth 4 +``` + +## User Table has duplicate user entry + +Carrisa Rimmer vs Carrissa Rimer + +![](../../.gitbook/assets/owl-dupe-carrissa.png) + +## ATM customer data with only a 88% match + +As you can see below, less than a 90% match in most cases is a false positive. Each dataset is a bit different, but in many cases you should tune your duplicates to roughly a 90+% match for interesting findings. + +![](../../.gitbook/assets/owl-dupes.png) + +## Simple DataFrame Example + +![](../../.gitbook/assets/owl-dupe-df.png) diff --git a/dq-visuals/more/explorer-2.md b/dq-visuals/more/explorer-2.md new file mode 100644 index 00000000..cb9f9489 --- /dev/null +++ b/dq-visuals/more/explorer-2.md @@ -0,0 +1,117 @@ +--- +description: A no-code option to get started quickly and onboard a dataset. +--- + +# Explorer (no-code) + +![](<../../.gitbook/assets/explorer (3).gif>) + +## Getting Started + +This page can be accessed by clicked the Explorer option (the compass icon). + +![](<../../.gitbook/assets/image (96).png>) + +{% hint style="info" %} +All UI functionality has corresponding API endpoints to define, run, and get results programmatically. +{% endhint %} + +## Select Your Data Source + +![](<../../.gitbook/assets/image (105).png>) + +## Create a new DQ Job by clicking +Create DQ Job + +![](<../../.gitbook/assets/image (126).png>) + +#### **View Data is an interactive option to run queries and explore the data** + +#### The bar chart icon will take you to a profile page of the dataset created prior to Explorer 2 + +## Select The Scope and Define a Query + +![](<../../.gitbook/assets/image (128).png>) + +#### Pick Date Column if your dataset contains an appropriate time filter + +#### Click Build Model -> to Save and Continue + +![](<../../.gitbook/assets/image (139).png>) + +## Transform Tab (advanced / optional) + +{% content-ref url="../../dq-job-examples/owlcheck/owlcheck-transform.md" %} +[owlcheck-transform.md](../../dq-job-examples/owlcheck/owlcheck-transform.md) +{% endcontent-ref %} + +#### Click Build Model -> to Save and Continue + +## Profile + +![](<../../.gitbook/assets/image (97).png>) + +#### Use the drop-downs to enable different analysis. Best practice is to leave the defaults. + +## Pattern (advanced / optional) + +Toggle on Pattern to enable this layer + +Click +Add to define a group and series of columns + +{% content-ref url="pattern-mining.md" %} +[pattern-mining.md](pattern-mining.md) +{% endcontent-ref %} + +#### Click Save to and Click Outlier to Continue + +## Outlier (advanced / optional) + +{% content-ref url="outliers.md" %} +[outliers.md](outliers.md) +{% endcontent-ref %} + +#### Click Save to and Click Dupe to Continue + +## Dupe (advanced / optional) + +{% content-ref url="duplicates.md" %} +[duplicates.md](duplicates.md) +{% endcontent-ref %} + +#### Click Save to and Click Source to Continue + +## Source (advanced / optional) + +Navigate to the source dataset + +Click Preview to interlace the columns + +Manually map the columns by dragging left to right or deselect columns + +{% content-ref url="validate-source.md" %} +[validate-source.md](validate-source.md) +{% endcontent-ref %} + +#### Click Save to and Click Save/Run to Continue + +## Run + +1. Select an agent +2. Click Estimate Job +3. Click Run to start the job + +![](<../../.gitbook/assets/image (116).png>) + +![](<../../.gitbook/assets/image (145).png>) + +{% hint style="info" %} +\*Note if you do not see your agent, please verify the agent has been assigned to your connection via: +{% endhint %} + +{% content-ref url="../../connecting-to-dbs-in-owl-web/add-connection-to-agent.md" %} +[add-connection-to-agent.md](../../connecting-to-dbs-in-owl-web/add-connection-to-agent.md) +{% endcontent-ref %} + +_Admin Console-->Remote Agent--> (Link icon on far right)-->Map connections to this agent and then reload the explorer page_ + +## **** diff --git a/dq-visuals/more/missing-records.md b/dq-visuals/more/missing-records.md new file mode 100644 index 00000000..bf87ffe0 --- /dev/null +++ b/dq-visuals/more/missing-records.md @@ -0,0 +1,17 @@ +# Records (advanced) + +{% hint style="info" %} +This is an advanced opt-in feature +{% endhint %} + +## Where'd my rows go? + +Owl is constantly learning which records or rows in a dataset are most common. In the case below the NYSE had a reasonable dataset volume (row count). + +![](../../.gitbook/assets/owl-missing-records.png) + +## Row Count Trend + +We can see the rows dipping just slightly outside their predicted range. Arguably a subtle drop, yet abnormal to not represent these companies that typically do trade on the NYSE. Were they de-listed? + +![](../../.gitbook/assets/owl-row-trend.png) diff --git a/dq-visuals/more/outliers.md b/dq-visuals/more/outliers.md new file mode 100644 index 00000000..edff67a9 --- /dev/null +++ b/dq-visuals/more/outliers.md @@ -0,0 +1,178 @@ +# Outliers (advanced) + +{% hint style="info" %} +This is an advanced opt-in feature +{% endhint %} + +## Numerical Outliers + +Kodak Coin! In 2018 Kodak announced themselves as Kodak coin and witnessed a steep change in their stock price. Owl automatically captured this event and provided the ability to drill into the item. + +![](<../../.gitbook/assets/owl-outlier-numerical (1).png>) + +### Complex outliers made Simple + +Even though Owl uses complex formulas to identify the correct outliers in a dataset, it uses simple terms when displaying them. If you notice below the change happened gradually, therefore if you only compared avgs or previous values you would not understand the full impact of this price change. 0% changed from yesterday and its moving/trailing avg would have caught up. + +![](<../../.gitbook/assets/owl-outlier-numerical (2).png>) + +## Dynamic history options + +Data may not always enter your data pipeline on time and as expected due to weekend, holidays, errors, etc. To help capture outliers in spite of gaps, there are two main options: + +* 1\) Extend the lookback period (to 10 days from 5 days, for example) +* 2\) Utilize additional flags per below (fllbminrow new as of 2021.11) + +| Flag | Description | Example | +| ---------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| fllbminrow | File Lookback Minimum Rows: determines minimum number of rows that a previous file scan needs to be counted as file lookback |

-fllbminrow 1 (counts nay DQ scans with 1 or more row in minimum history)

-fllbminrow 0 (default behavior, row count does not matter)

| +| dllb | Date Lookback: determines how many days of learning | -dllb 5 (5 days) | + +## Categorical Outliers + +Categorical Outliers are much different than numerical outliers and require separate techniques to automatically capture meaningful anomalies. The details regarding Owl's methodology and testing can be found below, 3 minute read on the topic. + +{% embed url="https://medium.com/owl-analytics/categorical-outliers-dont-exist-8f4e82070cb2" %} + +Owl will automatically learn the normal behavior of your String and Categorical attributes such as STOCK,OPTION,FUTURE or state codes such as MD,NC,D.C. When a strange pattern occurs (e.g NYC instead of NY), Owl will show this as a categorical outlier. + +Owl is able to detect Categorical Outliers both with and without taking time into account. If a time dimension is not provided, Owl will calculate the distribution of categorical values within the available data, and identify the values that fall into the most infrequent percentile (configurable). + +![Categorical Outliers without Time](<../../.gitbook/assets/Screen Shot 2020-07-07 at 9.43.19 PM.png>) + +If a time dimension is provided, Owl will first identify infrequent categories in the historical context and then in the context of the current Owlcheck. Only values that are historically infrequent or non-existent, and are infrequent in the current run will be considered Outliers. + +![Categorical Outliers with Time](<../../.gitbook/assets/Screen Shot 2020-07-07 at 9.37.17 PM.png>) + +## Training Outlier Detection Model + +Although Owl uses different techniques to detect Numerical and Categorical Outliers, the training process is very similar. + +At a minimum, Owl requires historical data that can be used as the training dataset. If no other input is provided, Owl will calculate the normal range for each selected column and look for numerical and categorical outliers within the training dataset without any further context. The output will essentially consist of infrequent values that fall outside the normal range fo each column. + +![](<../../.gitbook/assets/Screen Shot 2020-07-07 at 8.17.02 PM.png>) + +To obtain more targeted results, the Owl requires a "key" column. This column will be used to provide context by grouping each column by the key column. Defining a good key column tends to provide results that are a better indicators of actual data quality issues instead of simply infrequent values. + +![](<../../.gitbook/assets/Screen Shot 2020-07-07 at 8.18.40 PM.png>) + +Another input that can make outlier detection more precise is a data/time column and a look back period. This enables a more precise calculation of the normal range for a column and in the case of numerical outliers, makes it possible for Owl to establish a trend. Given a time column and key column, Owl will not only identify numerical outliers, it will plot the historical trend of the column value trailing the outlier. + +![](<../../.gitbook/assets/Screen Shot 2020-07-07 at 8.19.14 PM.png>) + +Owl also allows further refinement of the time dimension by defining time bins and processing intervals. By default, when given a time column, Owl will bin the data into days and process the data in daily interval. However, if the data is high frequency, day bins and day intervals might be too coarse grained. In this case, it might make more sense to group the data into bins on the minute and process the data in hour or minute intervals. The same concept applies in the other direction. What if the data is already aggregated on the month or year? In this case, it makes more sense to set the bins and intervals to month by month or month by year. + +![](<../../.gitbook/assets/Screen Shot 2020-07-07 at 8.20.18 PM.png>) + +Some data may be measured in really small or large units or contain a lot of noise. In this case, Owl allows the user to adjust the sensitivity level and unit of measure for outlier detection on each column. Click the advanced tab to make these adjustments. + +![](<../../.gitbook/assets/Screen Shot 2020-07-07 at 8.20.33 PM.png>) + +Once Outlier detection is complete for a given run, it's time to tune the scoring of the model. Owl allows the user to label any outlier findings as legitimate, thus preventing that outlier from being detected in the future or effecting the score of the current run. In addition, it is possible to define the significance of an outlier finding to a given dataset. This can be accomplished by setting how many quality points should be deducted for each outlier finding on any given run on that dataset. It is also possibly to adjust sensitivity and unit of measure of future runs by clicking on the small gear icon on the far left of the screen. + +![](<../../.gitbook/assets/Screen Shot 2020-07-07 at 8.38.05 PM.png>) + +## Spark DataFrame Example + +![](../../.gitbook/assets/owl-categorical-outlier.png) + +#### Real World Example + +Imagine you are the data manager at Iowa Department of Commerce, Alcoholic Beverage Division. As part of the Department's open data initiative, the monthly[ Iowa liquor sales data](https://data.iowa.gov/Sales-Distribution/Iowa-Liquor-Sales/m3tr-qhgy) are available to the public for analysis. (Thank you Iowa!) + +An Iowan data analyst emails you about a data quality issue with **address** for store #2508 in the year 2016. You quickly run a SQL query on your data warehouse to see what is going on. + +```sql +-- Assuming Postgres DB +select date_trunc('MONTH', "date") "date_month", address, count(*) "sales_count" +from iowa_liquor_sales +where "date" >= '2016-01-01' and "date" < '2017-01-01' and store_number = '2508' +group by date_trunc('MONTH', "date"), address +order by date_month, address +``` + +| date\_month | address | sales\_count | +| ------------------- | ------------------------- | ------------ | +| 2016-01-01 00:00:00 | 1843 JOHNSON AVENUE, N.W. | 422 | +| 2016-02-01 00:00:00 | 1843 JOHNSON AVENUE, N.W. | 451 | +| 2016-03-01 00:00:00 | 1843 JOHNSON AVENUE, N.W. | 579 | +| 2016-04-01 00:00:00 | 1843 JOHNSON AVENUE, N.W. | 404 | +| 2016-05-01 00:00:00 | 1843 Johnson Avenue, N.W. | 625 | +| 2016-06-01 00:00:00 | 1843 Johnson Avenue, N.W. | 695 | +| 2016-07-01 00:00:00 | 1843 Johnson Avenue, N.W. | 457 | +| 2016-08-01 00:00:00 | 1843 Johnson Avenue, N.W. | 744 | +| 2016-09-01 00:00:00 | 1843 Johnson Avenue, N.W. | 681 | +| 2016-10-01 00:00:00 | 1843 Johnson Avenue, N.W. | 728 | +| 2016-11-01 00:00:00 | 1843 Johnson Avenue, N.W. | 1062 | +| 2016-12-01 00:00:00 | 1843 Johnson Avenue, N.W. | 992 | + +Because `store_number` is an unique number assigned to the store who ordered the liquor, the inconsistent `address` values for the same store pose data quality problem. But `address` is a string value that can take many forms. For store #2508, the reported address value has a shifted behavior from all capital letters starting on May 2016. For other cases, it might be completely different behavior change that you would have to manually check one by one. With over 2,000 unique stores, 19 million rows, and 8 years of data, you need an automated way to detect meaningful categorical outliers. + +The following command shows an example of running monthly OwlDQ Checks, from the month of Jan 2016 to the month of December 2016. Each monthly run looks back 3 months of data to establish a baseline for categorical columns that you suspect would have similar data quality issues: `store_name`, `address`, and`city`. + +```bash +/opt/owl/bin/owlcheck + # connection information to data + -lib "/opt/owl/drivers/postgres/" -driver "org.postgresql.Driver" + -c, "jdbc:postgresql://localhost:5432/postgres" + -u, "postgres", "-p", "password" + # Specify dataset name + -ds "iowa_liquor_sales_by_store_number_monthly" + # Specify date filter for the last filter, e.g. date >= '2016-12-01' and date < '2017-01-01' + -rd "2016-12-01" -rdEnd "2017-01-01" + # SQL query template (${rd} and ${rdEnd} matches with -rd and -rdEnd + -q "select distinct on (date, store_number) date, store_number, store_name, address, city + from iowa_liquor_sales where date >= '${rd}' and date < '${rdEnd}' " + # Turn on Outliers + -dl + # Group on store_number (optional if no grouping) + -dlkey "store_number" + # Specify column that is of date type (optional, if running OwlCheck without time context) + -dc "date" + # Specify columns to run Outlier analysis (if not specified, all the columns in query are included in analysis) + -dlinc "store_name,address,city" + # Specify 3 month lookback for each OwlCheck + -dllb 3 + # Run Monthly OwlCheck + -tbin "MONTH" + # "backrun" Convenient way to run 12 preceding MONTHly owl check + -br 12 +``` + +**Results** + +The `-br 12` option ran 12 monthly OwlChecks for every month of 2016. The figure below shows OwlCheck Hoot page for the lastest run of dataset `iowa_liquor_sales_by_store_numbers_monthly`. The Hoot page shows that OwlCheck identified 24 Outliers among 4.8k rows of unique date x store\_number for month of December, 2016. + +![Monthly OwlCheck for 2016-12-01](<../../.gitbook/assets/image (39).png>) + +Since the original data quality issue that inspired us to run OwlCheck is from May 2016, we can navigate to specific run date 2016-05-01 by click on the line graph on top. Then searching for store #2508 on the **key** column shows outlier detected for **column** `address`. Press \[+] for that row to see contextual details about this detected value. + +![Monthly OwlCheck for 2016-05-01. The drill-in outlier details for store #2508 is shown](<../../.gitbook/assets/image (36).png>) + +We can verify that OwlCheck identified the outlier of interest among other 60 data quality issues. Using OwlCheck, you can identify issues at scale for past data (using backrun), current (using simple OwlCheck), and future (using scheduled jobs). + + + +## \*Tech Preview \[TP] Outlier Calibration\* + +**Use Case** + +When A) using Outliers and B) faced with an event such as a stock split or currency devaluation, it may be helpful to calibrate the outlier boundaries within Collibra DQ to avoid surfacing non-issues for a period of time. + +**Example Step #1: No Action Necessary** + +In the video below, Collibra DQ Outliers were set to a high sensitivity. The USDEUR conversion rate on January 6th in the sample dataset may be considered reasonable and the user can 1) rerun the dataset with lower sensitivity or 2) downtrain the unintended Outlier anomalies. + +![](../../.gitbook/assets/outliercalibratenavigate.gif) + +**Example Step #2: Macro Event That User Understands e.g. Currency Devaluation or Stock Split** + +When examining the outlier on January 11th, the dataset depicts that the USDEUR conversion shot up to 3.14, which in our hypothetical example coincides with an explainable macroeconomic phenomenon. As such, the user may not want Outlier anomalies to trigger for a period of time. + +![](../../.gitbook/assets/outliercalibratenextdatapoint.gif) + +**Example Step #3: User Wants To Suppress Outliers** + +Once Outlier Calibration is enabled, a user can select the boundaries and duration of the 'suppression' period. And once the DQ Job is re-run for the selected date(s), the outliers will not trigger an anomaly / downscore. + +![](../../.gitbook/assets/outliercalibratemodal.gif) diff --git a/dq-visuals/more/overview.md b/dq-visuals/more/overview.md new file mode 100644 index 00000000..09701947 --- /dev/null +++ b/dq-visuals/more/overview.md @@ -0,0 +1,79 @@ +# Summary + +## **Click or Code** + +Collibra DQ offers easy to use no (low) code options for getting started quickly. Alternatively, more technical users may prefer programmatic APIs. + +## **Core Components** + +Collibra DQ offers a full DQ suite to cover the unique challenges of each dataset. + +**9 Dimensions of DQ** + +1. Behaviors - Data observability +2. Rules - SQL-based rules engine +3. Schema - When columns are added or dropped +4. Shapes - Typos and Formatting Anomalies +5. Duplicates - Fuzzy matching, Identify similar but not exact entries +6. Outliers - Anomalous records, clustering, time-series, categorical +7. Pattern - Classification, cross-column & parent/child anomalies +8. Record - Deltas for a given column(s) +9. Source - Source to target reconciliation + +[Check out our videos to learn more](https://www.youtube.com/channel/UCKMcJ5NRiCDZQxBvSsVtTXw/videos) + +## **Behavior** + +**Imagine a column going null, automatic row count checks - does your data behave/look/feel the same way it has in the past.** + +![](../../.gitbook/assets/Behavior.jpg) + +## **Rules** + +**Assures only values compliant with your data rules are allowed within a data object.** + +![](../../.gitbook/assets/Rules.jpg) + +## **Schema** + +**Columns add or dropped.** + +![](../../.gitbook/assets/Schema.jpg) + +## **Shapes** + +**Infrequent formats.** + +![](<../../.gitbook/assets/shapes (1).jpg>) + +## Dupes + +**Fuzzy matching to identify entries that have been added multiple times with similar but not exact detail.** + +![](../../.gitbook/assets/Dupes.jpg) + +## **Outliers** + +**Data points that differ significantly from other observations.** + +![](../../.gitbook/assets/Outliers.jpg) + +## **Pattern** + +**Recognizing relevant patterns between data examples.** + +![](../../.gitbook/assets/Pattern.jpg) + +## **Source** + +**Validating source to target accuracy.** + +![](../../.gitbook/assets/Source.jpg) + +## **Record** + +**Deltas for a given column.** + +![](../../.gitbook/assets/Record.jpg) + +## diff --git a/dq-visuals/more/pattern-mining.md b/dq-visuals/more/pattern-mining.md new file mode 100644 index 00000000..aa489c60 --- /dev/null +++ b/dq-visuals/more/pattern-mining.md @@ -0,0 +1,23 @@ +# Patterns (advanced) + +{% hint style="info" %} +This is an advanced opt-in feature +{% endhint %} + +Owl uses the latest advancements in data science and ML to find deep patterns across millions of rows and columns. In the example below it noticed that Valerie is likely the same user as she has the same customer\_id and card\_number but recently showed up with a different last name. Possible misspelling or data quality issue? + +![](../../.gitbook/assets/owl-patterns.png) + +## Training Anti-Pattern Detection Model + +When the Patterns feature is enabled, Owl will build a collection of patterns that it identifies within the data. It will then use that collection to identify values that break established patterns. For example, in the image below, Owl learned that a bike route that starts at "MLK library" will end at "San Jose Diridon Caltrain Station". However, when the current day's data cross referenced against this pattern, Owl detects an anti-pattern where a trip starts at "MLK Library" but ends at "Market at 4th". Owl raises this anti-pattern as a data quality issue and highlights the what it believes the "end\_station" value should have been. + +To build a Pattern model, Owl requires historical data that contains the valid patterns and if possible, a date/time column. The user can then needs to define the date/time column, the look back period, and what columns make up the pattern. In the image below, the pattern was composed of "end\_station", "start\_terminal", "start\_station". + +It is possible that an apparent anti-pattern finding is actually valid data and not a data quality issue. In this case, Owl allows the user to further instruct the existing Patterns model on how to properly score and handle the findings. For example, if it turns out that "Market at 4th" is actually a valid "end\_station" for a bike trip, the user can negate the identified anti-pattern by labeling it as valid. This action will instruct Owl to not raise this particular anti-pattern again. Owl will also rescore the current Owlcheck results to reflect the user's feedback. In addition, it is possible to define the weight of an anti-pattern finding on the current dataset by setting the numerical value to deduct per finding. + +![](<../../.gitbook/assets/Screen Shot 2020-03-19 at 5.55.49 PM.png>) + +## Fraud Detection? + +Think about a scenario where a dataset has a SSN column along with FNAME, LNAME and many others. What if your traditional rules engine passes because one of the rows has a valid SSN and a valid Name but the SSN doesn't belong to that person (his or her name and address, etc...)? This is where data mining can derive more sophisticated insights than a rules based approach. diff --git a/dq-visuals/more/pushdown.md b/dq-visuals/more/pushdown.md new file mode 100644 index 00000000..f4a723fa --- /dev/null +++ b/dq-visuals/more/pushdown.md @@ -0,0 +1,59 @@ +--- +description: Pushdown feature description +--- + +# Pushdown + +Some of the DQ features support pushdown to avoid transferring large dataset from data source (Database, Cloud storage, file systems etc.) into Spark. When pushdown is enabled and supported, the DQ Job will generate SQL queries to offload the compute to the data source, reducing the amount of data transfer and Spark computation of the DQ Job. Not all features support pushdown nor pushdown completely eliminate data transfer. + +## Why use pushdown + +To analyze a dataset for DQ findings as part of a DQ Job, CDQ uses Spark as the compute engine to power the analysis. This requires the dataset to be loaded into Spark as a Spark DataFrame, and the DQ Job performance (job completion speed) is limited by Spark resources available and the complexity of the job. This data transfer from data source to Spark is dependent on two of the following factors: + +1. Bandwidth limitation at the data source\ + If the DQ Job requires pulling 100 million rows from a SQL Database, then the Input/Output limit of transferring data out of SQL Database into the Spark cluster will greatly increase the overall speed of the DQ Job. +2. Compute limitation at the data source for complex queries\ + If the DQ Job requires complex queries to create the dataset (via `-q`), then this computation is done at the SQL Database level. Some examples of "complex" queries are:\ + \ + `-q "SELECT * FROM public.very_long_table_of_transactions WHERE date = ${rd} and department = 'finance'"`\ + This query can take long time at the database level due to `WHERE` clause filtering a very long table on columns `date` and `department` that may not be indexed properly, leading to a full table scan without a `LIMIT` clause specified.\ + \ + `-q "SELECT * FROM public.very_long_table_of_transactions transactions INNER JOIN public.departments departments ON transactions.department_id = departiments.id WHERE transactions.date = ${rd} and departments.name = 'finance'"`\ + This query can take long time at the database level due to a join between tables. +3. Resource limitation at the data source\ + The data source may not have enough hardware resource available to efficiently fulfill the query and/or handle multiple DQ Jobs & other non-CDQ applications requesting data. + +The bottleneck due to data transfer can be viewed in Jobs page > Job Logs > LOAD stage. This data loading step is the first step for all DQ Jobs. However, pushdown feature can be used to _reduce_ (NOT eliminate) the data transfer & compute to Spark from the data source, if the DQ Job specified does not require loading all the data into Spark. \ +\ +In summary, speed of loading the data from data source to Spark is a `-q query compute time at data source` + `network transfer of -q result between data source and Spark`. Pushdown can reduce both elements, but the efficiency gained is dependent on the complexity of `-q` query and how big the dataset from `-q`result would have been without pushdown. + +## How pushdown is efficient + +Some of the Spark compute performed by the DQ Job can be translated into SQL queries that most relational databases support natively. In such a case, the DQ Job does not need to load all the rows of the dataset. Instead, the DQ Job can query the data source for the results of those SQL queries and reduce the amount of data transferring out of the data source. The results of these SQL queries are almost always lead to smaller amount of data compared to the full dataset defined by `-q`. Only some of the DQ Job features require the full dataset to be loaded into Spark. Therefore, pushdown can be a useful tool to speed up the overall DQ Job speed -- **provided that the speed of executing these SQL queries are faster than the speed of transferring the data out of the data source into Spark**. In most use cases, pushdown leads to faster DQ Job execution for large datasets. If the `-q` query is sufficiently complex, then the speed reduced by transferring less data into Spark can be cancelled out by the multiple frequent SQL queries made to the data source by the Pushdown process (because each query may have have redundant compute due to the complexity of `-q`) + +## How pushdown works in CDQ + +Using pushdown only _reduces_ the amount of data transferred out of the data source. It does NOT skip the LOAD stage in DQ Job. Every DQ Job requires a small sample of rows (10-20) of the dataset defined by `-q` in order to generate Data Preview and analyze schema information for the dataset run. This means the `-q` query may be fully computed at data source before the sampling can occur (depending on the complexity of the `-q`). In such a case, sampling 10-20 rows of data is not a quick and immediate LOAD stage and only efficiency gain comes from lack of transferring data between data source and Spark. \ +\ +Therefore, pushdown feature would be most efficient if `-q` is a simple select query with simple where filtering. The benefit comes from the fact that if your dataset defined by `-q` results in 100 million rows, only 10-20 rows of the dataset defined by `-q` will be loaded into Spark. + +Profile with pushdown will then generate a series of SQL queries and query the data source again for aggregate metric data. Depending on the dataset, these multiple SQL queries can be more efficient than loading all the data into Spark and computing these aggregate metric in Spark. The results between _Profile with pushdown_ and _Profile without pushdown_ are (practically) identical. + +## Profile: pushdown vs no pushdown + +Here is the summary of Profile activity with details regarding pushdown support + +| Feature | Supports pushdown | Description | +| -------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Row Count | Yes | Computes row count of the dataset. | +| Distinct Count | Yes | The number of distinct values in a column | +| Mean | Yes | The average of all the values in the column. Supports numeric columns only. | +| Min / Max | Yes | The minimum and maximum values of the column. Supports numeric and boolean columns only | +| NULL Count | Yes | The number of null values in the column. | +| EMPTY Count | Yes | The number of empty values in the column. Supports string columns only. | +| TYPE Count | Yes | The number of different types inferred in a column (if any) | +| TopN / BottomN | No |

Computes the top 5 most frequent (TopN) and least frequent (BottomN) values. Supports all types.

This result is displayed as a frequency bar chart in Profile page. If pushdown is enabled, then TopN and BottomN values are not displayed. Related features like Stat Rules (Distribution) are also disabled.

| +| Data Shape Detection | No | Detects shapes of the values based on Shape parameters provided (automatic or manual). | +| Histogram | No | Creates histogram of the values in the column. | +| Correlation Matrix | No | Creates correlation matrix. Supports only numeric columns | + diff --git a/dq-visuals/more/schema-evolution.md b/dq-visuals/more/schema-evolution.md new file mode 100644 index 00000000..912cdac4 --- /dev/null +++ b/dq-visuals/more/schema-evolution.md @@ -0,0 +1,13 @@ +--- +description: Detect schema evolution and unexpected schema changes. +--- + +# Schema (automatic) + +Dataset schemas are the columns or fields that define the dataset. They are often located in the header row of a tabular file or database table. However, JSON and XML are two examples of formats that include schema columns that are not in the header but rather nested throughout the document. OwlDQ automatically without needing to turn on any features detects the schema columns as well as reads or infers their data types (varchar, string, double, decimal, int, date, timestamp etc...). Owl observes each dataset so if a column is ever altered, removed or added it will automatically raise the event via its standard composite scoring system. + +#### Scoring... Alerting... Schema Detection... Automatically + +![](../../.gitbook/assets/owl-schema.png) + +Schema Evolution is one of Owl's 9 DQ dimensions. It can be an important measurement for data stewards to understand how the dataset is changing overtime. The orange bar on the chart shows a change in schema and allows for drilling in over time. diff --git a/dq-visuals/more/shapes.md b/dq-visuals/more/shapes.md new file mode 100644 index 00000000..b2f6f987 --- /dev/null +++ b/dq-visuals/more/shapes.md @@ -0,0 +1,17 @@ +# Shapes (automatic) + +Owl will automatically detect inconsistencies in data formats. These inconsistencies are where Data Scientists spend an enormous amount of time cleaning the data before building a ML model. Many reports have documented that over 80% of the time it takes to build a credible model comes from first understanding all the different formats and then writing munging or ETL style code to clean it before processing. What about all the patterns the process or person doesn't even know about? + +![](../../.gitbook/assets/owl-phone-shapes.png) + +### Drill-in to any Shape anomaly and see a visual example + +See an itemized list view of the most infrequent or odd shapes in your datasets. + +![](../../.gitbook/assets/owl-shape-drillin.png) + +### Shape Tuning + +By clicking the gear icon in the upper right corner of the SHAPE tab on the HOOT page. + +![](../../.gitbook/assets/shape-tuning-owl.png) diff --git a/dq-visuals/more/validate-source.md b/dq-visuals/more/validate-source.md new file mode 100644 index 00000000..a079ba32 --- /dev/null +++ b/dq-visuals/more/validate-source.md @@ -0,0 +1,49 @@ +# Source (advanced) + +{% hint style="info" %} +This is an advanced opt-in feature +{% endhint %} + +## Does your data-lake reconcile with your upstream system? + +Copying data from one system to another is probably the most common data activity to all organizations. Owl refers to this as source to target. As simple as this activity sounds, Owl has found that most of the time files and database tables are not being copied properly. To ensure and protect against target systems getting out of sync or not matching the originating source, turn on `-vs` to validate that the source matches the target. + +## A row count is not enough... + +The most common check we encounter is a row count. However, a row count does not account for: + +* Schema differences - Boolean to Int, Decimal to Double with precision loss, Timestamps and Dates +* Value differences - Char or Varchars with whitespace vs Strings, null chars, delimiter fields that cause shifting, and much more. + +![](<../../.gitbook/assets/Screen Shot 2019-10-01 at 8.50.39 PM.png>) + +## OwlCheck Created from Wizard + +The Owl Wizard GUI creates the below OwlCheck which it can execute from the GUI by clicking RUN or by pasting at the cmdline. + +```bash +-lib /home/ec2-user/owl/drivers/valdrivers \ +-driver org.postgresql.Driver \ +-u user -p password \ +-c "jdbc:postgresql://ec2-34-227-151-67.compute-1.amazonaws.com:5432/postgres" \ +-q "select * from public.dateseries4" \ +-ds psql_dateseries2 -rd 2018-11-07 \ +-srcq select dz, sym as symz, high as highz, low as lowz, close as closez, volume as volumez, changed as changedz, changep as changepz, adjclose as adjclosez, open as openz from lake.dateseries \ +-srcu user \ +-srcp password \ +-srcds mysqlSYMZ \ +-srcd com.mysql.cj.jdbc.Driver \ +-srcc "jdbc:mysql://owldatalake.chzid9w0hpyi.us-east-1.rds.amazonaws.com:3306/lake" \ -valsrckey "SYMZ" \ +-vs \ +-valsrcinc "dz,symz,openz,highz,lowz,closez,volumez,changedz,changepz,adjclosez" +``` + +### End of Day Stock Data from Oracle to Mysql + +In this example we loaded NYSE\_EOD data in both Oracle and Mysql and then used Owl's Source Validation feature. We see 3 main classes of issues. 1) The row count is off by 1 row, this means a row was dropped or went missing when the data was copied. 2) The schemas don't exactly match. 3) In 2 cases the values are different at the cell level. NULL vs NYSE and 137.4 vs 137.42 + +![](<../../.gitbook/assets/Screen Shot 2019-10-09 at 9.40.55 AM.png>) + +### Latest View in 2.13+ + +![](../../.gitbook/assets/source.png) diff --git a/dq-visuals/profile.md b/dq-visuals/profile.md new file mode 100644 index 00000000..cb4a2351 --- /dev/null +++ b/dq-visuals/profile.md @@ -0,0 +1,135 @@ +--- +description: Create profiles based on a table, view, or file +--- + +# Profile (automatic) + +{% hint style="info" %} +Users have the option to scan the entire dataset or users can apply custom filtering to select the depth (row filtering) and width (columns). +{% endhint %} + +### Select the Scope + +You can find detailed instructions about [selecting the scope](more/explorer-2.md#select-the-scope-and-define-a-query) in the Explorer section. You can run limits, by time, or full table scans if you have enough resources. + +![](../.gitbook/assets/profile\_scope.gif) + +### Select Options (or leave defaults) + +![](../.gitbook/assets/profile\_options.gif) + +### Save / Run + +![Profile is on by default and is part of onboarding a dataset](<../.gitbook/assets/profile\_setup (1).gif>) + +### View the Results + +![](<../.gitbook/assets/profile\_results (1).gif>) + +## Automatically Profile + +Owl automatically profiles datasets over time to enable drill-in for detailed insights an automated data quality. A profile is just the first step towards an amazing amount of auto discovery. Visualize segments of the dataset and how how the dataset is changing over time. + +Collibra DQ offers click or code options to run profiling. + +## Dataset Profile + +Owl creates a detailed profile of each dataset under management. This profile will later be used to both provide insight and automatically identify data quality issues. + +![](<../.gitbook/assets/Screen Shot 2020-07-08 at 12.45.19 AM.png>) + +### Pushdown Profiling + +Collibra DQ can compute the Profile of a dataset either via Spark (default) or the Data Warehouse (Profile Pushdown) where the data lives as the engine. When the Profile is computed using the datasource DBMS the user can choose two levels of pushdown: + +* Full Profile - Perform full profile calculation except for TopN +* Count - Only perform row and column counts + +![](../.gitbook/assets/pushdown.gif) + +{% hint style="info" %} +The following DBMS systems are supported for "Profile Pushdown" + +* Impala +* Hive +* Snowflake +* Presto +* Teradata +* SQL Server +* Postgres +* Redshift +* Mysql +* Oracle +* DB2 +{% endhint %} + +{% hint style="danger" %} +Pushdown and parallel JDBC cannot be used together. If you are using pushdown, do not select the parallel JDBC option. +{% endhint %} + +![](<../.gitbook/assets/Screen Shot 2020-05-07 at 7.28.25 PM.png>) + +## Profile Insights + +![](<../.gitbook/assets/Screen Shot 2020-05-07 at 7.33.16 PM.png>) + +By gathering a variety of different statistics, Owl's profile can provide a great deal of insight about the dataset. + +{% hint style="info" %} +Profile includes a range of statistics + +* Actual Datatype +* Discovered Datatypes +* Percent Null +* Percent Empty +* Percent Mixed Types +* Cardinality +* Minimum +* Maximum +* Mean +* TopN / BottomN +* Value Quartiles +{% endhint %} + +## Sensitive Data Detection (Semantic) + +Owl can automatically identify any types of common PII columns. + +{% hint style="info" %} +Owl is able to detect the following types of PII + +* EMAIL +* PHONE +* ZIP CODE +* STATE CD +* CREDIT CARD +* GENDER +* SSN +* IP ADDRESS +* EIN +{% endhint %} + +![](<../.gitbook/assets/Screen Shot 2020-07-08 at 12.37.10 AM.png>) + +Once detected, Owl will tag the column in the Profile as the discovered type as well as automatically apply a rule. If the user can choose to decline any discovered tag by simply clicking on it and confirming the delete action. This action can also remove the rule associated with the tag. + +![](<../.gitbook/assets/Screen Shot 2020-07-08 at 12.39.13 AM.png>) + +## Correlation Matrix (Relationship) + +Discover hidden relationships and measure the strength of those relationships. + +![](../.gitbook/assets/owl-relationships.png) + +## Histograms + +Often the first step in a data science project is to segment the data. Owl automatically does this using histograms. + +![](../.gitbook/assets/owl-histogram.png) + +## Data Preview + +After profiling the data, for those users with appropriate rights, Owl provides a glimpse of the dataset. The Data preview tab also provides a some basic insights such as highlights of Data Shape issues and Outliers (if enabled), and Column Filtergram visualization. + +![](<../.gitbook/assets/Screen Shot 2020-05-07 at 7.57.29 PM.png>) + diff --git a/dq-visuals/rules/README.md b/dq-visuals/rules/README.md new file mode 100644 index 00000000..51a0aff6 --- /dev/null +++ b/dq-visuals/rules/README.md @@ -0,0 +1,57 @@ +--- +description: Users can apply custom monitoring with SQL +--- + +# Rules (user-defined) + +### SQL Rule Engine + +![](../../.gitbook/assets/rules.gif) + +### Rules. Can't live with them, can't live without them. + +Owl takes a strong stance that data should first be profiled, auto-discovered and learned before applying basic rules. This methodology commonly removes thousands of rules that will never need to be written and evolve naturally overtime. However there are still many cases to add a simple rule, complex rule or domain specific rule. Simply search for any dataset and add a rule. You can use the optional Column Name/Category/Description to add meta-data to your rules for future reporting. + +![](<../../.gitbook/assets/quick\_rules (1).gif>) + +{% hint style="info" %} +Customized discovery routines can be run using the [rule library](./#rule-library) together with [data concepts and semantics.](data-concepts-and-semantics.md) +{% endhint %} + +### Query Builder + +Query builder will help generate SQL for more complex rules. You can apply to one or two tables (Table A on left and Table B on right). The query builder can help build up multi-part conditions. + +![](../../.gitbook/assets/query\_builder.gif) + +![(Optional) Start by searching for table B on the right, to set a key for the join condition](<../../.gitbook/assets/Screen Shot 2019-09-04 at 12.39.17 PM.png>) + +![Input conditions and click SQL statement to generate example syntax](<../../.gitbook/assets/Screen Shot 2019-09-04 at 12.46.02 PM.png>) + +{% hint style="info" %} +As with any SQL generator, there are limitations for more complex scenarios. +{% endhint %} + +### Break Records + +Storing break records is only available for Freefrom and Simple rule types. Rule library rules will use one of these types as well. + +![](../../.gitbook/assets/break\_records.gif) + +Additional storage can be enabled using the -linkid flag. This allows users to store complete sets of break records. Please refer to the [linkid section](../../dq-job-examples/owlcheck/owlcheck-linkid.md) for more details + +{% hint style="info" %} +Stat, Native, and Data Type (global) rules are not eligible for storing exception records. +{% endhint %} + +### Quick Tips + +If joining more than one data source, make sure both sets of drivers are in the -lib. Or separately supply a -libsrc pointing to the appropriate directory/jar file location. Version 2021.11 and forward will use the -addlib for additional directories to add to the classpath. + +Native SQL uses your native DB syntax. The score is total break records / rows from the scope (query / -q) of the defined DQ job. + +### Spark SQL + +This is a complete list of Spark SQL operators and functions available. [https://spark.apache.org/docs/latest/api/sql/index.html](https://spark.apache.org/docs/latest/api/sql/index.html) + +![](<../../.gitbook/assets/sql\_functions (1).gif>) diff --git a/dq-visuals/rules/creating-a-business-rule.md b/dq-visuals/rules/creating-a-business-rule.md new file mode 100644 index 00000000..2c5bd3af --- /dev/null +++ b/dq-visuals/rules/creating-a-business-rule.md @@ -0,0 +1,79 @@ +# Adding a Rule + +![](../../.gitbook/assets/rule\_.gif) + +Navigate to the Rule page as shown in the screenshot below. By selecting the icon on the top left corner under the Owl icon and select “Rule”. You will see a page as shown below. + +![](<../../.gitbook/assets/Screen Shot 2019-12-24 at 9.08.27 AM.png>) + +You can get to rules from the hoot page by clicking on Rules under the dataset name and score in the top right of the screen.‌ + +![](<../../.gitbook/assets/Screen Shot 2019-12-24 at 9.07.59 AM.png>) + +## Instructions + +1. Search for a dataset or navigate to the rule page using the hot link + * Rules can only be applied to datasets once a DQ job has been run +2. Click Load + * This will populate the schema and any previously saved rules +3. Select a rule type + * Using the dropdown next to the Type label +4. Select a rule name + * If applying a preset rule, the rule name will be auto populated +5. Input a rule condition + * Only if applying a simple, freeform sql, stat, or native rule type. + * Provide a value in the condition/sql/function input field. + * Keystroke Ctrl+Space will provide IntelliSense +6. Select Low, Medium or High for scoring severity (optional). +7. Add any custom DQ dimensions for reporting (optional). +8. Click submit to save the rule. + +![Search for a dataset and click Select next to the Type label](<../../.gitbook/assets/image (18) (5).png>) + +‌The rule will be measured on the next DQ job for that particular dataset.‌ + +#### **Rule Types** + +1. When to use a simple rule: + * Simple rules would be applied to filter a condition on a single column in a single table. + * Example: city = 'Baltimore' +2. When to use a freeform sql rule: + * ​Complex (freeform sql) would be used when applying a condition across multiple tables/columns and generally when more flexibility/customization is desired. + * ​Example: select \* from dataset where name = 'Owl' +3. When to use a preset rule: + * Preset rules would be used for quickly adding strict condition check. Commonly used conditions are available to add to any dataset columns.‌ + +All built-in spark functions are available to use. ([https://spark.apache.org/docs/2.3.0/api/sql/](https://spark.apache.org/docs/2.3.0/api/sql/)) for simple and freeform sql rules.‌ + +#### **Points and Percentage** + +For every percentage the X condition occurs, deduct Y points from the data quality score. If a rule was triggered 10 times out of 100 rows, break records occurred 10% of the time. If you input 1 point for every 1 percent, 10 points would be deducted from the overall score.‌ + +#### **Creating Your First Rule** + +Let’s create a simple rule using the below information. The dataset name. + +1. Search for “shape\_example” and click “Load” +2. Select “Simple Rule” +3. Rule Name = lnametest +4. @shape\_example.lname = “hootbeck” (should hit one time day over day). +5. Points = 1 +6. Percentage = 1 +7. Click “Submit” + +![](<../../.gitbook/assets/Screen Shot 2019-09-04 at 1.32.15 PM.png>) + +Once the rule has been submitted please find the below list of rules with the new rule we just defined as shown below. + +#### **Seeing Your First Rule Get Triggered** + +![](<../../.gitbook/assets/Screen Shot 2019-09-04 at 1.29.35 PM.png>) + +Rule scores will appear under the Rule tab on the Hoot page. You can also see more details in the bottom panel of the Rules page under the Rules and Results tabs. + +![Hoot page rule results](<../../.gitbook/assets/Screen Shot 2019-09-04 at 1.29.54 PM.png>) + +![Rule page rule results (bottom panel)](<../../.gitbook/assets/Screen Shot 2019-09-04 at 1.30.07 PM.png>) + + + diff --git a/dq-visuals/rules/data-concepts-and-semantics.md b/dq-visuals/rules/data-concepts-and-semantics.md new file mode 100644 index 00000000..59cb229f --- /dev/null +++ b/dq-visuals/rules/data-concepts-and-semantics.md @@ -0,0 +1,108 @@ +--- +description: >- + Custom data discovery and enforcement using rule templates (data concepts and + semantics) +--- + +# Rule Discovery + +## Data Concepts + +A data concept is the class or family of a dataset for example: _Loan Data, Stock Data, Position Limit Data, Retail Product Data, Patient Health Data, Interest Rate Data, etc..._ By giving data classes or "classifying" datasets we can transfer common understanding, rules and ML to datasets. This powerful technique allows a data steward to set up concepts once and enables the entire organization the ability to unify and standardize on common rules and terms, solving many metadata scale challenges. + +{% hint style="info" %} +Dataset Level + +**Security Reference Data** - Bloomberg financial data **Home Loan Data** - Mortgage application data +{% endhint %} + +## Semantics + +{% hint style="info" %} +Column Level + +EMAIL, ZIP CODE, SSN, CUSIP, GENDER, ADDRESS, CURRENCY CD, SKU, EIN, IP ADDRESS, PHONE, LICENSE, VIN, CREDIT CARD +{% endhint %} + +A semantic is the "semantic type" of a column or attribute of a dataset. All columns have a physical type such as String, Int, Date etc... but the semantic understanding of what type of String is in the column can be very important. I also allows us to enforce DQ validation rules out of the box. + +Owl's semantic scanning self identifies standard columns and automatically provides the proper protection. This makes it easy to get started adding common rules for specific use-cases. + +**Owl offers out of the box rules for 1-click rule creation** + +![](<../../.gitbook/assets/Auto Rules.png>) + +## Sensitive Data + +{% hint style="info" %} +Column Level + +**PII** - personally identifiable information **MNPI** - materially non public information\ +**PCI** - credit information like a credit card number **PHI** - HIPAA medical information +{% endhint %} + +## Data Discovery: The Power of Combining All Three into One Domain + +{% embed url="https://www.youtube.com/watch?v=wLXh71fn1l0" %} + +Now imagine if you could classify your datasets as concepts, then automatically have all the columns be recognized semantically(with validation rules in place) as well as have the columns labeled with sensitivity tags. It might look something like the below. + +![](<../../.gitbook/assets/Screen Shot 2021-09-15 at 1.11.06 PM.png>) + +## Steps To Use + +### Step 1: Create DQ Job with Semantic On + +To start, create a new DQ Job and select Semantic ON on the Profile options page + +![](<../../.gitbook/assets/dqjobsemanticon (1).gif>) + +### Step 2: In Catalog, select and apply your Data Concept + +Navigate to your dataset in **Catalog** and select the Data Concept you would like to apply with the **Actions** dropdown menu. + +See below sections on how to **Administering Data Concepts** as well as how to **Create and Manage Semantics** + +![](<../../.gitbook/assets/semantic-apply-dataconcept (1).gif>) + +### Step 3: Rerun your DQ Job with applied Data Concept + +Please rerun your DQ Job so that Collibra DQ can 1) profile your data, 2) auto-generate the rules based on the Semantics under the Data Concept, and 3) highlight any break records. + +![](../../.gitbook/assets/dataconcept-rerun-job.gif) + +### Success! Review Findings + +On the Profile page, please observe the newly tagged Semantics on the applicable columns + +On the DQ Job page, please browse your newly created rules based on Semantics as well as any corresponding rule breaks + +![](../../.gitbook/assets/review-dataconcept-results.gif) + +## Creating and Managing Semantics + +Create, test, and manage your Semantics in Collibra DQ in your **Rule Builder** wizard on the **Create Generic Rule** tab. Below is an example of creating a RegEx Semantic + +![](<../../.gitbook/assets/creating-semantic-regex (1).gif>) + +## Administering Data Concepts + +Setup your data concepts once and let the entire organization benefit by unifying all datasets to a common understanding in the admin Data Concepts page. + +![](../../.gitbook/assets/adding-data-concepts.gif) + +![](<../../.gitbook/assets/Screen Shot 2021-09-15 at 1.14.42 PM.png>) + +## Physical Schemas to Semantics + +Below you can see the benefit of organized metadata. PDEs or `physical data elements` organized/tagged by semantics. This allows for sub-second searches while in catalog or searching for data to figure out where all your PII data lives, or what systems have "loan data". + +![](<../../.gitbook/assets/Screen Shot 2021-09-15 at 4.32.09 PM.png>) + +Above you can see Data Concepts in Yellow, Semantics in Gray and Sensitive labels in Orange. Enabling you to organize all your data in classes, search and discover types no matter what system they live in or what the PDE column name is. Transforming technical types into business metadata. + +## Business Unit Roll up Reporting + +Now that we have all PDEs discovered and tagged and rolled up into business terms, we can roll up technical assets like database tables and files into business reports across departments and non technical concepts. + +![](<../../.gitbook/assets/Screen Shot 2021-09-15 at 5.17.14 PM.png>) diff --git a/dq-visuals/rules/dimensions.md b/dq-visuals/rules/dimensions.md new file mode 100644 index 00000000..0a69ebc4 --- /dev/null +++ b/dq-visuals/rules/dimensions.md @@ -0,0 +1,5 @@ +# Dimensions + +Each rule can be associated to a dimension. You can also apply metadata like primary column and description. While not mandatory, these can be particularly when using the [DQ Connector](../../integration/dq-connector.md) and [custom reporting](../../reports/custom.md). + +![](../../.gitbook/assets/rule\_dimensions.gif) diff --git a/dq-visuals/rules/dq-rule-automation.md b/dq-visuals/rules/dq-rule-automation.md new file mode 100644 index 00000000..06bd5e86 --- /dev/null +++ b/dq-visuals/rules/dq-rule-automation.md @@ -0,0 +1,50 @@ +--- +description: An organized repository of all your rule templates +--- + +# Rule Library + +![](../../.gitbook/assets/rule\_lib.gif) + +#### The rule library contains both OOTB and custom-built rule templates. + +![](<../../.gitbook/assets/image (95).png>) + +### OOTB Rules + +Owl shares all of it's out of the box rules with each user/tenant. This makes it easy to get started quickly and let the team add common rules for specific use-cases. See Rule Library for more information. + +Below is a list of one click rules that can be added to any dataset. It is important to note that Owl often self identifies these columns and automatically provides the proper protection. + +* Email +* Zip +* Credit Card +* SSN +* EIN +* State Code +* Phone +* Gender +* IP Address +* Date +* Int +* Double + +The quick rule drop-down offers another great way to apply rules at the click of a button in the preview tab. + +![](<../../.gitbook/assets/image (41).png>) + +### Customized Rules + +#### Add to The Rule Library + +Create a rule once using our rule template builder and re-use the rule across any column on any dataset. Owl will substitute the dataset and column that the rule applies to at runtime. This commonly saves hundreds of redundant rules that do the same thing but on different column names. + +The rule library hosts out of the box and custom global rules. See data concepts and semantics for advanced use of global rules. + +{% content-ref url="data-concepts-and-semantics.md" %} +[data-concepts-and-semantics.md](data-concepts-and-semantics.md) +{% endcontent-ref %} + +#### + +### diff --git a/dq-visuals/rules/rule-templates.md b/dq-visuals/rules/rule-templates.md new file mode 100644 index 00000000..b4d23e06 --- /dev/null +++ b/dq-visuals/rules/rule-templates.md @@ -0,0 +1,21 @@ +--- +description: Ideal for rules that apply to more than one dataset. Write once, apply many. +--- + +# Rule Templates + +![](<../../.gitbook/assets/rule\_template (1).gif>) + +Templates are great for regex, format, and compliance checks. + +A template rule will substitute the dataset and column at runtime. This commonly saves hundreds of redundant rules that do the same thing but on different column names. + +Templated rules can be found in the 'type' dropdown as well as the 'quick rules' drop-down. The complete list of template rules is located in the Rule Library section. These meant for global rules that are ideal for code sets, compliance checks, and regex checks. These are ideal for checks that apply to many tables. + +Rule templates will utilized SQLG (simple) and occasionally SQLF (Freeform) types under the hood. Rule templates will appear in the Rule Library after they are created. + +See the [rule library section](dq-rule-automation.md) for more details. + +{% hint style="info" %} +Customized discovery routines can be run using the [rule library](rule-templates.md#rule-library) together with [data concepts and semantics.](data-concepts-and-semantics.md) +{% endhint %} diff --git a/dq-visuals/rules/rule-types/README.md b/dq-visuals/rules/rule-types/README.md new file mode 100644 index 00000000..6ab29d78 --- /dev/null +++ b/dq-visuals/rules/rule-types/README.md @@ -0,0 +1,65 @@ +# Rule Types + +## SQL-Based Rules + +Depending on the complexity, users can choose from short form or long form rules. + +#### **Simple** + +Just the condition (short form). For example, using the column email\_address. This runs against the dataframe and uses Spark SQL syntax. An easy way to think about Simple rules is 'everything after the where clause'. + +``` +email_address is not null and email_address != '' +``` + +#### Freeform + +Where 'Simple' rules just use the condition, 'Freeform' rules use the complete SQL statement. When more complex SQL is required, you can express more with Freeform including joins, CTE's and window statements. + +``` +select * from @DATASET_NAME where +email_address is not null and email_address != '' +``` + +{% hint style="info" %} +All built-in spark functions are available to use. ([https://spark.apache.org/docs/2.3.0/api/sql/](https://spark.apache.org/docs/2.3.0/api/sql/)) for simple and freeform sql rules.‌ +{% endhint %} + +#### Native + +Native rules use the SQL dialect of the underlying connection and database. Files are not eligible for native SQL rules. This is ideal if you want to use push-down profiling and you want to use existing SQL logic. When coupled with push-down profiling, you can achieve a very minimal infrastructure footprint. + +See the [native rules section](sql-based-rules/native-sql.md) for more details. + +## Stat + +Write rules against meta data and profiling stats. Complex counts and ratios can be referenced with simple syntax. + +See the[ stat rules section ](stat-rules/)for more details. + +### Data Type + +* **Empty check** + * Rule type: EMPTYCHECK + * Description: Checking whether the target column has empty values or not +* **Null check** + * Rule type: NULLCHECK + * Description: Checking whether the target column has _NULL_ values or not +* **Date check** + * Rule type: DATECHECK + * Description: Checking whether the target column has only DATE values or not +* **Integer check** + * Rule type: INTCHECK + * Description: Checking whether the target column has only INTEGER values or not +* **Double check** + * Rule type: DOUBLECHECK + * Description: Checking whether the target column has only DOUBLE values or not +* **String check** + * Rule type: STRINGCHECK + * Description: Checking whether the target column has only STRING values or not. +* **Mixed datatype check** + * Rule type: DATATYPECHECK + * **Description:** --- + + + diff --git a/dq-visuals/rules/rule-types/sql-based-rules/README.md b/dq-visuals/rules/rule-types/sql-based-rules/README.md new file mode 100644 index 00000000..9ae62a1d --- /dev/null +++ b/dq-visuals/rules/rule-types/sql-based-rules/README.md @@ -0,0 +1,3 @@ +# SQL Based Rules + +Collibra DQ uses ANSI compliant SQL and even offers push-down rules that leverages specific database syntax. diff --git a/dq-visuals/rules/rule-types/sql-based-rules/freeform/README.md b/dq-visuals/rules/rule-types/sql-based-rules/freeform/README.md new file mode 100644 index 00000000..e4c75ac3 --- /dev/null +++ b/dq-visuals/rules/rule-types/sql-based-rules/freeform/README.md @@ -0,0 +1,28 @@ +--- +description: Fully defined SQL for more detailed checks +--- + +# Freeform + +Where 'Simple' rules just use the condition, 'Freeform' rules use the complete SQL statement. When more complex SQL is required, you can express more with Freeform including joins, CTE's and window statements. + +``` +select * from @DATASET_NAME where +email_address is not null and email_address != '' +``` + +{% hint style="danger" %} +Freeform rules should include a semi-colon at the end of the rule value. +{% endhint %} + +{% hint style="info" %} +All built-in spark functions are available to use. ([https://spark.apache.org/docs/2.3.0/api/sql/](https://spark.apache.org/docs/2.3.0/api/sql/)) for simple and freeform sql rules.‌ +{% endhint %} + +### Cross-Connection Libraries + +![](<../../../../../.gitbook/assets/image (143).png>) + +{% hint style="warning" %} +When applying cross-connection rules please use the -addlib to submit the job with the appropriate jar files. In this example, a secondary set of jars is added through the Explorer. These files are located in the /opt/owl/drivers/mysql directory. The path should not contain double quotes or single quotes. It should point to a directory without spaces in the path. +{% endhint %} diff --git a/dq-visuals/rules/rule-types/sql-based-rules/freeform/easier-rule-examples.md b/dq-visuals/rules/rule-types/sql-based-rules/freeform/easier-rule-examples.md new file mode 100644 index 00000000..6b86b95f --- /dev/null +++ b/dq-visuals/rules/rule-types/sql-based-rules/freeform/easier-rule-examples.md @@ -0,0 +1,62 @@ +# Cross-Dataset Rules + +{% hint style="danger" %} +**Best practice for cross-table joins is to define a view and scan the view.** + +**** + +**Only in circumstances when a view cannot be created should you define cross-table joins with 2 separate datasets (DQ Jobs) and express the join in the rule.** + +**** + +**If you're doing multiple lookups this will improve long-term performance and consolidate maintenance.** +{% endhint %} + +{% hint style="warning" %} +Cross-dataset rules require -libsrc or -addlib (prior to 2021.11) +{% endhint %} + +![When a cross-dataset rule uses two connections, be sure the jars are in the -lib or -addlib directory.](<../../../../../.gitbook/assets/image (134).png>) + +## In-Clause (Single Column) + +```sql +select * from @table1 where id +not in ( select id from @table2 ) +``` + +## Except (Multi-Column) + +```sql +select id, app_id, email, guid_num from @table1 +EXCEPT +select id, app_id, email, guid_num from @table2 +``` + +## Join Example + +```sql +SELECT + * +FROM + @table1 A + LEFT JOIN @table2 B ON A.id = B.id +where + B.id is null OR (A.email != B.email) +``` + +## Cross-Table (Guided). Use our wizard to do ad-hoc analysis and visual setup. + +{% embed url="https://www.youtube.com/watch?v=uQ0tilvBUKc" %} + +## Join Example Example (vs. cross-table guided seen above). + +{% embed url="https://www.youtube.com/watch?v=BFw5ZIzwewQ" %} + +## Multi-part condition rules with the rule builder. Combines profiling metrics & builder in one screen. + +{% embed url="https://www.youtube.com/watch?v=Zkw23umvf8o&feature=youtu.be&t=16" %} + +## Sample Results + +![](<../../../../../.gitbook/assets/image (46).png>) diff --git a/dq-visuals/rules/rule-types/sql-based-rules/native-sql.md b/dq-visuals/rules/rule-types/sql-based-rules/native-sql.md new file mode 100644 index 00000000..8f6553e9 --- /dev/null +++ b/dq-visuals/rules/rule-types/sql-based-rules/native-sql.md @@ -0,0 +1,28 @@ +--- +description: >- + With Native SQL expressions we provide capability to use your existing + validation statements, even if they are using database specific + (Postgre/DB2/Oracle/MSSQL/etc...) functions or expressions. +--- + +# Native + +This is commonly used when using pushdown profiling and/or pre-existing SQL validations exist in a specific syntax. Native rules are often referred to as pushdown rules. + +![](<../../../../.gitbook/assets/image (113).png>) + +If you have rules already written in Oracle, Sybase, or DB2 syntax - Copy/Paste the rule directly into the Native SQL section. + +When creating a Native SQL rule, keep the following in mind: + +* The SQL query must be a valid expression that can be run as a subquery. To avoid pulling large amounts of data into memory, Collibra DQ will wrap your expression so it only fetches the number of rows returned. Rules should be written such that the query returns the anomalous rows. +* The SQL query must take less than 30 minutes to run. It is recommended to use partitioned columns for efficiency. +* When testing the SQL query from the app, it is helpful if it take less than 30 seconds to run. You can add a limit to reduce query time, or test the query in your SQL Editor. + +{% hint style="info" %} +Testing native rules can be done quickly by limiting the results or using an external SQL IDE as well. +{% endhint %} + +{% hint style="warning" %} +At this rule type you won't have mismatch highlighting and record breakdown on the UI, just the score itself! +{% endhint %} diff --git a/dq-visuals/rules/rule-types/sql-based-rules/simple.md b/dq-visuals/rules/rule-types/sql-based-rules/simple.md new file mode 100644 index 00000000..ac573531 --- /dev/null +++ b/dq-visuals/rules/rule-types/sql-based-rules/simple.md @@ -0,0 +1,19 @@ +--- +description: Just the condition after the "where" clause +--- + +# Simple + +Depending on the complexity, users can choose from short form or long form rules. + +#### **Simple** + +Just the condition (short form). For example, using the column email\_address. This runs against the dataframe and uses Spark SQL syntax. An easy way to think about Simple rules is 'everything after the where clause'. + +``` +email_address is not null and email_address != '' +``` + +{% hint style="info" %} +All built-in spark functions are available to use. ([https://spark.apache.org/docs/2.3.0/api/sql/](https://spark.apache.org/docs/2.3.0/api/sql/)) for simple and freeform sql rules.‌ +{% endhint %} diff --git a/dq-visuals/rules/rule-types/stat-rules/README.md b/dq-visuals/rules/rule-types/stat-rules/README.md new file mode 100644 index 00000000..cee001c6 --- /dev/null +++ b/dq-visuals/rules/rule-types/stat-rules/README.md @@ -0,0 +1,56 @@ +# Stat Rules + +One really powerful technique is to access the profile statistics in your rules. These are typically sub-second operations that do not require scanning or iterating. There are several cases where SQL struggles to support rules, such as: isNull but not "null count" or nullRatio or nullPercent. Or having access to types without doing crazy cast() operations. These are simplified below, i.e. fname.$type == 'String' + +```sql +select * from @dataset where +fname.$type != 'String' AND $rowCount < 800 +``` + +| Dataset Level Stat | Rule Example | Description | +| ----------------------- | ------------------------ | ---------------------------------------------- | +| **$totalTimeInSeconds** | $totalTimeInSeconds > 25 | alert when DQ job runs longer than 25 seconds. | +| **$totalTimeInMinutes** | $totalTimeInMinutes > 5 | alert when DQ job runs longer than 5 mins. | +| **$totalTimeInHours** | $totalTimeInHours > 1 | alert when DQ job runs longer than 1 hour. | +| **$rowCount** | $rowCount < 9000 | alert when row count less than 9,000 | +| **$runId** | $runId = '2020-01-24' | use the ${rd} variable in rules | + +| Column Level Stat | Rule Example | Description | +| ------------------------------ | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | +| **.$type** | fname.$type != 'String' | alert when fname is not a string | +| **.$min** | fname.$min > 'apple' | lexicographical sort works for strings and numbers | +| **.$minNum** | age.$minNum > 13 | type casted to a numeric for simple number checks | +| **.$max** | fname.$max > 'apple' | | +| **.$maxNum** | age.$maxNum > 13 | | +| .**$uniqueCount** | id.$uniqueCount != $rowCount | alert when the uniqueCount of a field doesn't match the rowCount | +| **.$uniqueRatio** | gender.$uniqueRatio between .4 and .6 | | +| **.$nullRatio** | lname.$nullRatio not between .4 and .6 | alert when the ratio of nulls no longer falls within acceptable range | +| **.$nullPercent** | lname.$nullPercent not between 40 and 60 | alert when the percent of nulls no longer falls within acceptable range | +| **.$nullCount** | lname.$nullCount >= 1 | test for a single null | +| **.$emptyRatio** | nc.$emptyRatio > 0.2 | alert when the ratio of empties no longer falls within acceptable range | +| **.$emptyPercent** | nc.$emptyPercent > 20 | alert when the percent of empties no longer falls within acceptable range | +| **.$emptyCount** | | | +| **.$mixedTypeRatio** | nc.$mixedTypeRatio > 0.2 | **** | +| **.$mixedTypePercent** | nc.$mixedTypeRatio > 20 |

alert when the percent of mixed data types

no longer falls within acceptable range.

i.e. Strings and Ints in the same field

| +| **.$mixedTypeCount** | id.$mixedTypeCount >= 1 | | + +![](../../../../.gitbook/assets/colStatRules.png) + +Known limitation. Cannot combine stat rules or distribution rules with regex rules in the same rule. Example car\__vin rlike '$\[asdf]\[0-9]' and car\_vin.$uniqueCount_ + +### Distribution Rule + +There is a common case in DQ where you want to know the distribution of a columns value. Meaning if you have gender there may be an expectation that 40-60% are males and 40-60% are females if the dataset is large and represents the population. This can be rather hard or expensive to express in vanilla SQL, but is very easy with the below syntax. + +```sql +gender['Male'].$uniquePercent between 40 and 60 +``` + +| Column Value Level | Rule | +| ------------------- | ------------------------------------------------------- | +| **.$uniqueCount** | credit\_rating\['FAIR'].$uniqueCount > 7 | +| **.$uniquePercent** | credit\_rating\['GOOD'].uniquePercent between 40 and 60 | + +![](<../../../../.gitbook/assets/Screen Shot 2021-05-10 at 2.24.51 PM.png>) + +### diff --git a/dq-visuals/rules/rule-types/stat-rules/data-type-rules.md b/dq-visuals/rules/rule-types/stat-rules/data-type-rules.md new file mode 100644 index 00000000..645bdb40 --- /dev/null +++ b/dq-visuals/rules/rule-types/stat-rules/data-type-rules.md @@ -0,0 +1,38 @@ +# Data Type Rules + +## Data Type + +### Empty check + +**Rule type: EMPTYCHECK**\ +**Description:** Checking whether the target column has empty values or not + +### Null check + +**Rule type: NULLCHECK**\ +**Description:** Checking whether the target column has _NULL_ values or not + +### Date check + +**Rule type: DATECHECK**\ +**Description:** Checking whether the target column has **only** DATE values or not + +### Integer check + +**Rule type: INTCHECK**\ +**Description:** Checking whether the target column has **only** INTEGER values or not + +### Double check + +**Rule type: DOUBLECHECK**\ +**Description:** Checking whether the target column has **only** DOUBLE values or not + +### String check + +**Rule type: STRINGCHECK**\ +**Description:** Checking whether the target column has **only** STRING values or not. + +### Mixed datatype check + +**Rule type: DATATYPECHECK**\ +**Description:** --- diff --git a/dq-visuals/rules/scores.md b/dq-visuals/rules/scores.md new file mode 100644 index 00000000..7b815f43 --- /dev/null +++ b/dq-visuals/rules/scores.md @@ -0,0 +1,9 @@ +# Scores + +Each rule definition can have a specific score. Depending on the severity, rules can be hard or soft rules and can contribute a different weight that contributes to your overall DQ score. + +![](../../.gitbook/assets/rule\_scoring.gif) + +{% hint style="info" %} +This is calculated by points per threshold. The ratio is calculated from the total rows of the associated DQ job scan. +{% endhint %} diff --git a/installation/agent-configuration.md b/installation/agent-configuration.md new file mode 100644 index 00000000..a5a361c8 --- /dev/null +++ b/installation/agent-configuration.md @@ -0,0 +1,156 @@ +--- +description: DQ Agent Configuration Guide +--- + +# Agent + +## How to Install a New DQ Agent + +### Setting up a DQ Agent using `setup.sh` as part of DQ package + +Use `setup.sh` script located in `/opt/owl/` (or other [**Base Path**](https://docs.owl-analytics.com/installation/agent-configuration#agent-configuration-parameters) that your installation used). See example code block for installing a DQ Agent with Postgres server running `localhost` on port `5432` with database `postgres` and Postgres username/password combo `postgres`/`password` + +``` +# PATH TO DIR THAT CONTAINS THE INSTALL DIR +export BASE_PATH=/opt + +# PATH TO AGENT INSTALL DIR +export INSTALL_PATH=/opt/owl + +# DQ Metadata Postgres Storage settings +export METASTORE_HOST=localhost +export METASTORE_PORT=5432 +export METASTORE_DB=postgres +export METASTORE_USER=postgres +export METASTORE_PASSWORD=password + +cd $INSTALL_PATH + +# Install DQ Agent only +./setup.sh \ + -owlbase=$BASE_PATH \ + -options=owlagent \ + -pguser=$METASTORE_USER \ + -pgpassword=$METASTORE_PASSWORD \ + -pgserver=${METASTORE_HOST}:${METASTORE_PORT}/${METASTORE_DB} +``` + +The setup script will automatically generate the `/opt/owl/config/owl.properties` file and encrypt the provided password. + +### Setting up a DQ Agent manually + +* Passwords to DQ Metadata Postgres Storage should be encrypted before being stored in `/opt/owl/config/owl.properties`file. + +``` +# PATH TO AGENT INSTALL DIR +export INSTALL_PATH=/opt/owl + +cd $INSTALL_PATH + +# Encrypt DQ Metadata Postgres Storage password +./owlmanage.sh encrypt=password +``` + +`owlmanage.sh` will generate an encrypted string for the plain text password input. The encrypted string can be used in the `/opt/owl/config/owl.properties`configuration file to avoid exposing the DQ Metadata Postgres Storage password. + +* To complete Owl Agent configuration, edit the `/opt/owl/config/owl.properties`configuration file with basic agent values: + +``` +vi $INSTALL_PATH/config/owl.properties +``` + +* and add the following properties + +``` +spring.datasource.url=jdbc:postgresql://{DB_HOST}:{DB_PORT}/{METASTORE_DB} +spring.datasource.username={METASTORE_USER} +spring.datasource.password={METASTORE_PASSWORD} +spring.datasource.driver-class-name=com.owl.org.postgresql.Driver + +spring.agent.datasource.url=jdbc:postgresql://{DB_HOST}:{DB_PORT}/{METASTORE_DB} +spring.agent.datasource.username={METASTORE_USER} +spring.agent.datasource.password={METASTORE_PASSWORD} +spring.agent.datasource.driver-class-name=org.postgresql.Driver +``` + +* Restart the web app + +## How To Configure Agent via UI + +* Login to DQ Web and navigate to Admin Console. + +![Fig 1: Home Page](../.gitbook/assets/DQ-Admin-Console-1.png) + +* From the Admin Console, click on the Remote Agent tile. + +![Fig 2: Admin Console](../.gitbook/assets/DQ-Admin-Console-2.png.png) + +* Identify the row with the agent to edit . + +![Fig 3: Agent Management Table](../.gitbook/assets/DQ-Admin-Console-3.png) + +* Click on the pencil icon to edit. + +![Fig 4: DQ Agent with default values](../.gitbook/assets/DQ-Admin-Console-4.png) + +## How To Link DB Connection to Agent via UI + +When you add a new[ Database Connection](https://docs.owl-analytics.com/connecting-to-dbs-in-owl-web/owl-db-connection#how-to-add-db-connection-via-ui), the DQ Agent must be given the permission to run DQ Job via the specified agent. + +From Fig 3, select the chain link icon next to the DQ Agent to establish link to DB Connection. A modal to add give that agent permission to run DQ Jobs by DB Connection name will show (Fig 5). The left-side panel is the list DB Connection names that has not been linked to the DQ Agent. The right-side panel is the list of DB Connection names that has the permission to run DQ Job. + +Double click the DQ Connection name to move from left to right. In Fig 5, DB Connection named "metastore" is being added to DQ Agent. Click the "Update" button to save the new list of DB Connections. + +![Fig 5: Adding DB Connection named "metastore" to the DQ Agent](<../.gitbook/assets/Screenshot 2021-06-14 at 5.04.25 PM.png>) + +![Fig 6: How to add all connections to the selected DQ Agent](../.gitbook/assets/DQ-Agent-Config-Hint-1.png) + +{% content-ref url="../connecting-to-dbs-in-owl-web/add-connection-to-agent.md" %} +[add-connection-to-agent.md](../connecting-to-dbs-in-owl-web/add-connection-to-agent.md) +{% endcontent-ref %} + +## Agent Configuration Parameters + +| Parameter | Description | +| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **Is Local** | For Hadoop only | +| **Is Livy** | Deprecated. Not used. | +| **Base Path** |

The installation folder path for DQ. All other paths in DQ Agent are relative to this installation path

This is the location that was set as OWL_BASE in Full Standalone Setup and other installation setups followed by owl/ folder.

For example, if setup command was export OWL_BASE=/home/centos then the Base Path in the Agent configuration should be set to /home/centos/owl/.

Default: /opt/owl/

| +| **Owl Core JAR** |

The file path to DQ Core jar file.
Default <Base Path>/owl/bin/

| +| **Owl Core Logs** |

The folder path where DQ Core logs are stored. Logs from DQ Jobs are stored in this folder.
Default: <Base Path>/owl/log

| +| **Owl Web Logs** |

The folder path where DQ Web logs are stored. Logs from DQ Web App is stored in this folder.
Default: <Base Path>/owl/log

| +| **Owl Script** |

The file path to DQ execution script owlcheck.sh. This script is used to run DQ Job via command line without using agent. Usingowlcheck.shfor running DQ Jobs is superseded by DQ Agent execution model.
Default: <Base Path>/owl/bin/owlcheck

| +| **Deploy Deployment Mode** | The Spark deployment mode that takes one of `Client` or `Cluster` | +| **Default Master** | The Spark Master URL copied from the Spark cluster verification screen (`spark://...`) | +|

Default Queue

| The default resource queue for YARN | +| **Dynamic Spark Allocation** | Deprecated. Not used. | +| **Spark Conf Key** | Deprecated. Not used. | +| **Spark Conf Value** | Deprecated. Not used. | +| **Number of executor(s)** | The default number of executors allocated per DQ Job when using this Agent to run DQ Scan | +| **Executer Memory (GB)** | The default RAM per executors allocated per DQ Job when using this Agent to run DQ Scan | +| **Number of Core(s)** | The default number of cores per executors allocated per DQ Job when using this Agent to run DQ Scan | +| **Driver Memory (GB)** | The default driver RAM allocated per DQ Job when using this Agent to run DQ Scan | +|

Free Form (Appended)

| Other `spark-submit` parameters to append to each DQ Job when using this Agent to run DQ Scan | + +![Fig 2: Edit DQ Agent mode in D](<../.gitbook/assets/Screenshot 2021-06-14 at 4.25.09 PM.png>) + +## Setting up an HA Group + +If you have multiple DQ Agents, then you can establish them as an HA Group. When doing so, make sure both DQ Agents have the same connections established to them. + +* Click on the "AGENT GROUPS (H/A)" Tab name your HA Group and add the Agents you'd like to participate as Group. NOTE: HA GROUPS will execute jobs in a round robin fashion. + +![](https://gblobscdn.gitbook.com/assets%2F-L\_xJcI5Uc0S1x9JC6xQ%2F-LnUsmCFG-tKOAMRkqvx%2F-LnV8ur7dy\_09xGAIsA8%2FScreen%20Shot%202019-08-29%20at%209.18.40%20PM.png?alt=media\&token=92b1ecfb-b8a0-4f9c-ad6d-7d22601cafd5) + +* When the Agents have been registered, associated with DB connections, users can now execute a job via the explorer page. + +![Fig 7: Executing an Ad Hoc job via DQ Web Explorer](https://gblobscdn.gitbook.com/assets%2F-L\_xJcI5Uc0S1x9JC6xQ%2F-LnUsmCFG-tKOAMRkqvx%2F-LnVCneYAUxjqA84p5Fq%2FScreen%20Shot%202019-08-29%20at%209.45.42%20PM.png?alt=media\&token=06e9d717-d75d-430c-8a2c-a68720189a78) + +### Diagram + +![Fig 1: High level depiction of DQ Agents using CDH, HDP, and EMR within a single DQ Web App](https://gblobscdn.gitbook.com/assets%2F-L\_xJcI5Uc0S1x9JC6xQ%2F-LnU88TjMSmNDQQOzmga%2F-LnUBuNZqRfEFAzVhB0o%2FAgents%20\(1\).jpg?alt=media\&token=3452698c-aeae-43e4-b730-b2b19e4dd1c5) + +Fig 1 shows provides a high level depiction of how agents work within DQ. A job execution is driven by DQ Jobs that are written to an `agent_q` table inside the DQ Metadata Postgres Storage (`Owl-Postres` database in Fig 1) via the Web UI or REST API endpoint. Each agent available and running queries the `Owl-Postgres` table every 5 seconds to execute the DQ Jobs the agent is responsible for. For example, the EMR agent `Owl-Agent3` in Fig 1 only executes DQ Jobs scheduled to run on EMR. + +When an agent picks up a DQ Job to execute, the agent will launch the job either locally on the agent node itself or on a cluster as a spark job (if the agent is setup as an edge node of a cluster). Depending on where the job launches, the results of the DQ Job will write back to the DQ Metadata Storage (`Owl-Postgres` database). The results are then displayed on the DQ Web UI, exposed as REST API, and available for direct SQL query against `Owl-Postgres` database. + diff --git a/installation/cloud-native-owldq/README.md b/installation/cloud-native-owldq/README.md new file mode 100644 index 00000000..d5f14263 --- /dev/null +++ b/installation/cloud-native-owldq/README.md @@ -0,0 +1,63 @@ +# Cloud native + +## Introduction to cloud native architecture + +According to the Cloud Native Computing Foundation (“CNCF”) Charter: + +Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach. + +These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil. + +Collibra Data Quality wholeheartedly embraces these principles in its design and deployment. The diagram below depicts Collibra DQ's cloud native deployment architecture: + +![](../../.gitbook/assets/owl-k8s-deployment.png) + +In this "form factor", you can deploy Collibra DQ in any public or private cloud while maintaining a consistent experience, performance, and management runbook. + +### Collibra DQ microservices + +To achieve cloud native architecture, Collibra DQ is decomposed into several components, each of which is deployed as a microservice in a container. + +* **Owl Web** - The main point of entry and interaction between Collibra DQ and end users or integrated applications. Owl Web provides both a rich, interactive user experience and a robust set of APIs for automated integration. +* **Owl Agent** - You can think of the Agent as the "foreman" of Collibra DQ. When a user or application requests a data quality check through Owl Web, Owl Agent will marshal compute resources to perform the work. Owl Agent does not actually do any of the data quality work. Instead, it translates the request submitted by Owl Web into a technical descriptor of the work that needs to be done and then launches the requested [DQ job](broken-reference). +* **Owl Metastore** - This is where Collibra DQ stores all the metadata, statistics, and results of DQ jobs. It is also then main point of communication between Owl Web and Owl Agent. The metastore also contains the results of DQ jobs performed by transient containers (workers) in the compute space. +* **History Server** - Collibra DQ relies on Apache Spark to actually scan data and perform the bulk of data quality activities. To facilitate troubleshooting and performance tuning of DQ jobs, Collibra DQ uses an instance of Spark History Server to enable easy access to Spark logs. +* **Spark** - Apache Spark is the distributed compute framework that powers the Collibra DQ data quality engine. Spark enables DQ jobs to rise to the task of data quality on Terabyte scale datasets. Spark containers are completely ephemeral and only live for as long as necessary to complete a given DQ job. + +### Containerization + +The binaries and instruction sets described in each of the Collibra DQ microservices are encompassed within Docker container images. Each of the images is versioned and maintained in a secured cloud container registry repository. To initiate a Collibra DQ cloud native deployment, you must first obtain credentials to either pull the containers directly or download them to a private container registry. + +{% hint style="warning" %} +Support for Collibra DQ cloud native deployment is limited to deployments using the containers provided from the Collibra container registry. + +Reach out to your customer contact for access to pull the Collibra containers. +{% endhint %} + +### Kubernetes + +Kubernetes is a distributed container scheduler and has become synonymous with cloud native architecture. While Docker containers provide the logic and runtime at the application layer, most applications still require network, storage, and orchestration between multiple hosts in order to function. Kubernetes provides all of these facilities while abstracting away all of the complexity of the various technologies that power the public or private cloud hosting the application. + +### Collibra DQ Helm chart + +While Kubernetes currently provides the clearest path to gaining the benefits of a cloud native architecture, it is also one of the more complex technologies in existence. This has less to do with Kubernetes itself and more with the complexity of the constituent technologies it is trying to abstract. Technologies like attached distributed storage and software defined networks are entire areas of specialization that require extensive expertise to navigate. Well implemented Kubernetes platforms hide all of this complexity and make it possible for anyone to leverage these powerful concepts. However, a robust application like Collibra DQ requires many descriptors (K8s manifests) to deploy its various components and all of the required supporting resources like network and storage. + +This is where Helm comes in. Helm is a client side utility (since v3) that automatically generates all the descriptors needed to deploy a cloud native application. Helm receives instructions in the form of a **Helm chart** that includes templated and parameterized versions of Kubernetes manifests. Along with the Helm chart, you can also pass arguments like names of artifacts, connection details, enable and disable commands, and so on. Helm resolves the user defined parameters within the manifests and submits them to Kubernetes for deployment. This enables you to deploy the application without necessarily having a detailed understanding of the networking, storage or compute that underpins the application. + +For example, the command below deploys Collibra DQ with all of the components depicted in the [cloud native deployment architecture](./#introduction-to-cloud-native-architecture) diagram into Google Kubernetes Engine with Google Cloud Storage (GCS) as the storage location for Spark logs. The only perquisite is that the image pull secret, representing credentials to access the container registry, and secret containing the credentials for a service account with access to GCS are already deployed to the namespace. + +``` +helm upgrade --install --namespace \ +--set global.version.owl= \ +--set global.version.spark= \ +--set global.configMap.data.license_key= \ +--set global.spark_history.enabled=true \ +--set global.spark_history.logDirectory=gs://logs/spark-history/ \ +--set global.cloudStorage.gcs.enableGCS=true \ + \ +/path/to/chart/owldq +``` + +{% hint style="info" %} +The full universe of possible customizations is quite extensive and provides a great deal of flexibility in order to be applicable in a wide variety of platforms. However, when deploying on a known platform (EKS, GKE, AKS), the number of required inputs is quite limited. In common cases, you run a single CLI command including basic parameters like disable history server, configure the storage bucket for logs, specify the image repository, and so on. +{% endhint %} diff --git a/installation/cloud-native-owldq/aks-eks-gke-kubernetes-deployment.md b/installation/cloud-native-owldq/aks-eks-gke-kubernetes-deployment.md new file mode 100644 index 00000000..bfdd84c4 --- /dev/null +++ b/installation/cloud-native-owldq/aks-eks-gke-kubernetes-deployment.md @@ -0,0 +1,13 @@ +# EKS / GKE / AKS + +For organizations that are familiar with containers and managed Kubernetes services, Collibra DQ offers a cloud native deployment option. Please refer to the Cloud Native section for more detail. + +[Cloud Native](./) + +[Prerequisites](preparing-for-deployment.md) + +[Installation](deploying-cloud-native-owldq.md) + +{% hint style="info" %} +The containers are part of a private repository which requires access. Please contact Collibra directly for more information. +{% endhint %} diff --git a/installation/cloud-native-owldq/deploying-cloud-native-owldq.md b/installation/cloud-native-owldq/deploying-cloud-native-owldq.md new file mode 100644 index 00000000..5df2bead --- /dev/null +++ b/installation/cloud-native-owldq/deploying-cloud-native-owldq.md @@ -0,0 +1,131 @@ +# Cloud native install + +## Deploy Collibra DQ + +Once you have created a namespace and added all of the required secrets, you can begin the deployment of Collibra DQ. + +### Minimal install + +Install Web, Agent, and metastore. Collibra DQ is inaccessible until you manually add an Ingress or another type of externally accessible service. + +{% hint style="warning" %} +All of the following examples will pull containers directly from the Collibra DQ secured container registry. In most cases, InfoSec policies require that containers are sourced from a private container repository controlled by the local Cloud Ops team. Make sure to to add `--set global.image.repo=
` so that you use only approved containers. +{% endhint %} + +```bash +helm upgrade --install --namespace \ +--set global.version.owl= \ +--set global.version.spark= \ +--set global.configMap.data.license_key= \ +--set global.web.service.type=ClusterIP \ + \ +/path/to/chart/owldq +``` + +{% hint style="info" %} +The metastore container must start first as the other containers use it to write data. On your initial deplyment, the other containers might start before the metastore and fail. +{% endhint %} + +### Externally accessible service + +Perform the [minimal install](deploying-cloud-native-owldq.md#minimal-install) and add a preconfigured NodePort or LoadBalancer service to provide access to the Web. + +{% hint style="warning" %} +A LoadBalancer service type requires that the Kubernetes platform is integrated with a Software Defined Network solution. This will generally be true for the Kubernetes services offered by major cloud vendors. Private cloud platforms more commonly use Ingress controllers. Check with the infrastructure team before attempting to use LoadBalancer service type. +{% endhint %} + +```bash +helm upgrade --install --namespace \ +--set global.version.owl= \ +--set global.version.spark= \ +--set global.configMap.data.license_key= \ +--set global.web.service.type= \ + \ +/path/to/chart/owldq +``` + +#### Externally accessible with SSL enabled + +Perform the install with external service but with SSL enabled. + +{% hint style="info" %} +Ensure you have already deployed a keystore containing a key to the target namespace with a secret name that matches the `global.web.tls.key.secretName` argument (_owldq-ssl-secret_ by default). Also, ensure that the secret's key name matches the `global.web.tls.key.store.name` argument (_dqkeystore.jks_ by default). +{% endhint %} + +```bash +helm upgrade --install --namespace \ +--set global.version.owl= \ +--set global.version.spark= \ +--set global.configMap.data.license_key= \ +--set global.web.service.type= \ +--set global.web.tls.enabled=true \ +--set global.web.tls.key.secretName=owldq-ssl-secret \ +--set global.web.tls.key.alias= \ +--set global.web.tls.key.type= \ +--set global.web.tls.key.pass= \ +--set global.web.tls.key.store.name=keystore.jks \ + \ +/path/to/chart/owldq +``` + +#### Externally accessible and History Server for GCS Log Storage + +Perform the install with external service and Spark History Server enabled. In this example, the target log storage system is GCS. + +```bash +helm upgrade --install --namespace \ +--set global.version.owl= \ +--set global.version.spark= \ +--set global.configMap.data.license_key= \ +--set global.web.service.type= \ +--set global.spark_history.enabled=true \ +--set global.spark_history.logDirectory=gs://logs/spark-history/ \ +--set global.spark_history.service.type= \ +--set global.cloudStorage.gcs.enableGCS=true \ + \ +/path/to/chart/owldq +``` + +#### Externally Accessible and History Server for S3 Log Storage + +Perform the install with external service and Spark History Server enabled. In this example, the target log storage system is S3. + +{% hint style="info" %} +For Collibra DQ to be able to write Spark logs to S3, makes sure that an Instance Profile IAM Role with access to the log bucket is attached to all nodes serving the target namespace. +{% endhint %} + +```bash +helm upgrade --install --namespace \ +--set global.version.owl= \ +--set global.version.spark= \ +--set global.configMap.data.license_key= \ +--set global.web.service.type= \ +--set global.spark_history.enabled=true \ +--set global.spark_history.logDirectory=s3a://logs/spark-history/ \ +--set global.spark_history.service.type= \ +--set global.cloudStorage.s3.enableS3=true \ + \ +/path/to/chart/owldq +``` + +#### Externally accessible with external metastore + +Perform the install with external service and an external metastore, for example AWS RDS, Google Cloud SQL, or just PostgresSQL on its own instance. + +{% hint style="warning" %} +Collibra DQ currently supports PostgreSQL 9.6 and newer. +{% endhint %} + +```bash +helm upgrade --install --namespace \ +--set global.version.owl= \ +--set global.version.spark= \ +--set global.configMap.data.license_key= \ +--set global.web.service.type= \ +--set global.metastore.enabled=false +--set global.configMap.data.metastore_url=jdbc:postgresql://:/ +--set global.configMap.data.metastore_user= \ +--set global.configMap.data.metastore_pass= \ + \ +/path/to/chart/owldq +``` diff --git a/installation/cloud-native-owldq/preparing-for-deployment.md b/installation/cloud-native-owldq/preparing-for-deployment.md new file mode 100644 index 00000000..d2b3344e --- /dev/null +++ b/installation/cloud-native-owldq/preparing-for-deployment.md @@ -0,0 +1,178 @@ +# Cloud native requirements + +## Minimum requirements + +You need a machine with the following files and packages to run the installation. You can run these from a laptop or separate VM and they do not need to be issued on the Kubernetes cluster itself. + +### Prerequisites + +* [Helm](https://helm.sh). +* [kubectl](https://kubernetes.io/docs/tasks/tools/). +* Cloud command line SDK, such as gcloud CLI, AWS CLI or similar. + +### Files + +* The helm chart.\* +* JKS files with secrets created in kubectl: + * _owldq-ssl-secret_ + * _owldq-pull-secret_\* +* A _spark-gcs-secret_ you create from your service account file or token. + +{% hint style="info" %} +\* Available upon request from Collibra. +{% endhint %} + +### Application system requirements + +| Component | Processor | Memory | Storage | +| ------------- | --------- | ------ | ---------- | +| Owl Web | 1 core | 2 GB | 10 MB PVC | +| Owl Agent | 1 core | 1 GB | 100 MB PVC | +| Owl Metastore | 1 core | 2 GB | 10 GB PVC | +| Spark\* | 2 cores | 2 GB | - | + +{% hint style="info" %} +\* This is the minimum quantity of resources required to run an a Spark job in Kubernetes. This amount of resources would only provide the ability to scan a few megabytes of data with no more than a single job running at a given time. Proper sizing of the compute space must take into account the largest dataset that may be scanned, as well as the desired concurrency. +{% endhint %} + +## Network service considerations + +Owl Web is the only required component that needs to be directly accessed from outside of Kubernetes. History Server is the only other component that can be accessed directly by users, however, it is optional. + +If the target Kubernetes platform supports a LoadBalancer service type, you can configure the Helm chart to directly deploy the externally accessible endpoint. + +{% hint style="info" %} +For testing purposes, you can also configure the Helm chart to deploy a NodePort service type. +{% endhint %} + +For the Ingress service type, deploy OwlDQ without an externally accessible service and then attach the Ingress service separately. This applies when you use a third-party Ingress controller such as NGINX, Contour, and so on. + +{% hint style="info" %} +The Helm chart is able to deploy an Ingress on GKE and EKS platforms, however, there is a wide variety of possible Ingress configurations that have not been tested. +{% endhint %} + +## Obtaining credentials + +Kubernetes stores credentials in the form of secrets. Secrets are base64 encoded files that you can mount into application containers and that application components can reference at runtime. You use _pull secrets_ to access secured container registries to obtain application containers. + +### SSL certificates + +To enable SSL for secure access to Owl Web, a keystore that contains a signed certificate, keychain, and private key is required. This keystore must be available in the target namespace before you deploy Collibra DQ. + +{% hint style="info" %} +By default, Collibra DQ looks for a secret called `owldq-ssl-secret` to find the keystore. +{% endhint %} + +{% hint style="info" %} +Although it is possible to deploy with SSL disabled, is not recommended. +{% endhint %} + +### Cloud storage credentials + +If you enable History Server, a distributed filesystem is required. Currently, Collibra DQ supports S3 and GCS for Spark history log storage. + +{% hint style="info" %} +Azure Blob and HDFS on the near term roadmap. +{% endhint %} + +| Target storage system | Credentials requirements | +| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| S3 | An IAM Role with access to the target bucket needs to be attached to the Kubernetes nodes of the namespace where OwlDQ is being deployed. | +| GCS |

You must create a secret from the JSON key file of a service account with access the the log bucket.
The secret must be available in the namespace before you deploy Collibra DQ. By default, Collibra DQ looks for a secret called spark-gcs-secret, if GCS is enabled for Spark history logs. You can change this via a helm chart argument.

| + +### Container pull secret + +Collibra DQ containers are stored in a secured repository in Google Container Registry. For Collibra DQ to successfully pull the containers when deployed, a pull secret with access to the container registry must be available in the target namespace. + +{% hint style="info" %} +By default, Collibra DQ looks for a pull secret named `owldq-pull-secret`. You can change this via a helm chart argument. +{% endhint %} + +### Spark service account + +To anable Owl Agent and the Spark driver to create and destroy compute containers, you must have a service account with a role that allows get/list/create/delete operations on pods/services/secrets/configMaps in the target namespace. By default, OwlDQ attempts to create the required service account and the required RoleBinding to the default Edit role. Edit is a role that is generally available in a Kubernetes cluster. If the Edit role is not available, you can manually create it. + +## Accessing the platform + +To deploy anything to a Kubernetes cluster, the first step is to install the required client utilities and configure access: + +* **kubectl**: The main method of communication with a Kubernetes cluster. All configuration or introspection tasks will be preformed using kubectl. +* **helm v3**: Used to deploy the OwlDQ helm chart without hand coding manifests. + +After you install the utilities, the next step is to configure a kube-context that points to and authenticates to the target platform. On cloud platforms like GKE and EKS, this process is completely automated through their respective CLI utilities. + +``` +aws eks --region update-kubeconfig --name +``` + +``` +gcloud container clusters get-credentials +``` + +In private clouds, this process will vary from organization to organization, however, the platform infrastructure team should be able to provide the target kube-context entry. + +## Preparing secrets + +Once access to the target platform is confirmed, you can begin the preparation of the namespace. Typically the namespace that Collibra DQ is going to be deployed into is pre-allocated by the platform team. + +``` +kubectl create namespace +``` + +{% hint style="info" %} +There is a lot more that can go into namespace creation such as resource quota allocation, but that is generally a task for the platform team. +{% endhint %} + +### Create an SSL keystore secret + +``` +kubectl create secret generic owldq-ssl-secret \ +--from-file /path/to/keystore.jks \ +--namespace +``` + +{% hint style="warning" %} +The file name that you use in the `--from-file` argument should be _keystore.jks_. If the file name is anything else, you must include an additional argument specifying the keystore file name in the Helm command. +{% endhint %} + +### Create a container pull secret + +{% tabs %} +{% tab title="JSON key file credential" %} +``` +kubectl create secret docker-registry owldq-pull-secret \ +--docker-server= \ +--docker-username=_json_key \ +--docker-email= \ +--docker-password="$(cat /path/to/key.json)" \ +--namespace +``` +{% endtab %} + +{% tab title="Short lived access token" %} +``` +kubectl create secret docker-registry owldq-pull-secret \ +--docker-server= \ +--docker-username=oauth3accesstoken \ +--docker-email= \ +--docker-password="" \ +--namespace +``` + +{% hint style="warning" %} +GCP Oauth tokens are usually only good for 1 hour. This type of credential is excellent if the goal is to pull containers into a private registry. It can be used as the pull secret to access containers directly, however, the secret would have to be recreated with a fresh token before restarting any of the Colbra DQ components. +{% endhint %} +{% endtab %} +{% endtabs %} + +### Create a GSC credential secret + +``` +kubectl create secret generic spark-gcs-secret \ +--from-file /path/to/keystore.jks \ +--namespace +``` + +{% hint style="warning" %} +The file name that youuse in the `--from-file` argument should be _spark-gcs-secret_. If the file name is anything else, you must include an additional argument specifying the gcs secret name in the Helm command. +{% endhint %} diff --git a/installation/hadoop/README.md b/installation/hadoop/README.md new file mode 100644 index 00000000..250b9f45 --- /dev/null +++ b/installation/hadoop/README.md @@ -0,0 +1,5 @@ +# Hadoop + +For large scale processing and concurrency, a single vertically scaled Spark server is not enough. To address large scale processing, DQ has the ability to push compute to an external Hadoop cluster. This page describes the process by which the DQ Agent can be configured to push DQ jobs to Hadoop. + +![](<../../.gitbook/assets/Screenshot 2021-06-21 at 9.05.39 AM.png>) diff --git a/installation/hadoop/emr.md b/installation/hadoop/emr.md new file mode 100644 index 00000000..44530b96 --- /dev/null +++ b/installation/hadoop/emr.md @@ -0,0 +1,101 @@ +# EMR / Dataproc / HDI + +Running Apache Spark on Kubernetes differs from running this on virtual machine-based Hadoop clusters, which is the current mechanism provided by the existing CloudProc Dataproc service or competitive offerings like Amazon Web Services (AWS) Elastic MapReduce (EMR) and Microsoft's Azure HDInsight (HDI). + +Each cloud provider will have unique steps and configuration options. More detail on enabling agents for this deployment option be found in the [Hadoop Integration](https://docs.owl-analytics.com/installation/hadoop-integration) section. + +A detailed guide for EMR is provided below. + +## OwlDQ on EMR Architecture + +OwlDQ is able to use EMR as the compute space for data quality jobs (Owlchecks). While it is possible to simply operate a long running EMR cluster, EMR's intended operation model is ephemeral infrastructure. Using EMR as an ephemeral compute space is the most cost effective approach both in terms of operational effort and infrastructure costs. OwlDQ makes it possible to seamlessly leverage EMR in this operating model. When there is not an EMR cluster available, OwlDQ users are still able to browse datasets and DQ results in Web. However, if a user attempts to deploy an Owlcheck, they will simply see a red light icon next to the target agent. If the user still wants to request an Owlcheck, it will simply wait in queue until the target agent comes back online the next time an EMR cluster is available. + +![](<../../.gitbook/assets/Screenshot 2021-02-17 at 9.47.05 AM.png>) + +### Prepare for Deployment + +{% hint style="info" %} +Before enabling OwlDQ to use EMR as the compute space, make sure that Owl Web and the Owl Metastore are already deployed ([https://docs.owl-analytics.com/installation/full-install](https://docs.owl-analytics.com/installation/full-install)). +{% endhint %} + +1. Create a "bootstrap bucket" location in S3 where OwlDQ binaries and bootstrap script (install-agent-emr.sh) will be staged. The EMR cluster instances will need to include an attached Role that has access to this location in order to bootstrap the cluster. This location should not contain any data or any kind of sensitive materials and thus should not require any special permissions. It just needs to be accessible by EMR clusters for bootstrap purposes. +2. Create or modify an instance Profile Role that will be attached to EMR clusters so that it enables read access to the bootstrap bucket location. This Role is separate from the EMR service role that EMR will use to deploy the infrastructure. +3. Stage the bootstrap script and the OwlDQ binary package in the bootstrap location created above. +4. Make sure that the VPC where the Owl Metastore is deployed is accessible from the VPC where EMR clusters will be deployed. +5. Make sure that Security Groups applied to the Owl Metastore are configured to allow access from EMR master and worker Security Groups. +6. Decide whether to use EMR 5.x or EMR 6.x. This is important because EMR 6 introduces Spark 3 and Scala 2.12. If EMR 6 is chosen, make sure OwlDQ binaries were compiled for Spark 3 and Scala 2.12. +7. (OPTIONAL) Create and store a private key to access EMR instances + +### Deploy EMR Cluster + +There are several ways to deploy EMR, however, for dev-ops purposes, the simplest path is usually to use the AWS CLI utility. The example below will deploy and EMR cluster bootstrapped with OwlDQ binaries and a functioning agent to deploy Owlchecks. + +{% hint style="info" %} +When defining the Bootstrap Location argument, do not include "s3://". For example: + +If Bootstrap Location is s3://bucket/prefix then BOOTSTRAP\_LOCATION="bucket/prefix" +{% endhint %} + +``` +aws emr create-cluster \ +--auto-scaling-role EMR_AutoScaling_DefaultRole \ +--applications Name=Hadoop Name=Spark Name=Hive Name=Tez \ +--name owl-emr \ +--release-label emr-6.2.0 \ +--region ${EMR_REGION} \ +--ebs-root-volume-size 10 \ +--scale-down-behavior TERMINATE_AT_TASK_COMPLETION \ +--enable-debugging \ +--bootstrap-actions \ +"[{\"Path\":\"s3://${BOOTSTRAP_LOCATION}/install-agent-emr.sh\", \ +\"Args\":[ \ +\"${OWL_VERSION}\", \ +\"${OWL_AGENT_ID}\", \ +\"${METASTORE_HOST}:${METASTORE_PORT}/${METASTORE_DB}?currentSchema=owlhub\", \ +\"${METASTORE_USER}\", \ +\"${METASTORE_PASSWORD}\", \ +\"${BOOTSTRAP_LOCATION}\", \ +\"${LICENSE_KEY}\", \ +\"native\"], \"Name\":\"install-owl-agent\"}]" \ +--ec2-attributes "{ \ +\"KeyName\":\"${EMR_INSTANCE_PRIVATE_KEY_NAME}\", \ +\"InstanceProfile\":\"${BOOTSTRAP_ACCESS_ROLE}\", \ +\"SubnetId\":\"${EMR_SUBNET}\", \ +\"EmrManagedSlaveSecurityGroup\":\"${EMR_WORKER_SECURITY_GROUP}\", \ +\"EmrManagedMasterSecurityGroup\":\"${EMR_WORKER_SECURITY_GROUP}\" \ +}" \ +--service-role ${EMR_SERVICE_ROLE} \ +--log-uri s3n://${EMR_LOG_LOCATION} \ +--instance-groups "[ \ +{\"InstanceCount\":1,\"InstanceGroupType\":\"MASTER\",\"InstanceType\":\"${EMR_MASTER_INSTANCE_TYPE}\",\"Name\":\"Master - 1\"}, \ +{\"InstanceCount\":3,\"InstanceGroupType\":\"CORE\",\"InstanceType\":\"${EMR_CORE_INSTANCE_TYPE}\",\"Name\":\"Core - 2\"} \ +]" +``` + +### Configure Agent + +Once the EMR cluster and Owl Agent is deployed, it needs to be configured in Owl Web. + +1. Log into Owl Web, click the gear icon in the Navigation Pane and select "Admin Console". +2. In the Admin Console, click on the "Remote Agent" tile. +3. The newly created agent should have a green light icon. + +![](<../../.gitbook/assets/Screenshot 2021-02-17 at 10.53.19 AM.png>) + +Click the Pen/Pencil icon to the far right to configure the agent's settings. Make sure that Deploy Mode is set to "Cluster" and Default Master is set to "yarn". + +![](<../../.gitbook/assets/Screenshot 2021-02-17 at 10.57.39 AM.png>) + +Click the chain link icon to the far right to configure what datasources the agent is able to deploy Owlchecks for. + +![](<../../.gitbook/assets/Screenshot 2021-02-17 at 11.00.48 AM.png>) + +Any datasources that are not listed in the right hand pane will not be visible to this agent. + +### Run Owlchecks + +Everything is now ready for users to use EMR to run Owlchecks on data. Review Explorer documentation for detailed instructions. + +{% content-ref url="../../dq-visuals/more/explorer-2.md" %} +[explorer-2.md](../../dq-visuals/more/explorer-2.md) +{% endcontent-ref %} diff --git a/installation/hadoop/hadoop-integration.md b/installation/hadoop/hadoop-integration.md new file mode 100644 index 00000000..f0940c72 --- /dev/null +++ b/installation/hadoop/hadoop-integration.md @@ -0,0 +1,137 @@ +# Hadoop Install + +{% hint style="info" %} +In some cases, the required Hadoop client configuration requires the DQ Agent to run on an Hadoop Edge node within the cluster. This can happen because native dependency packages are required, network isolation from subnet that is hosting DQ server, complex security configuration, ect. In these circumstances, simply deploy the DQ Agent on a cluster Edge Node that contains the required configurations and packages. In this setup, the DQ Agent will use the existing Hadoop configuration and packages to run DQ checks on the Hadoop cluster. +{% endhint %} + +## Hadoop Config Setup + +Hadoop configuration can be incredibly complex. There can be hundreds of "knobs" across dozens of different components. However, DQ's goal is to simply leverage Hadoop to allocate compute resources in order to execute DQ checks (Spark jobs). This means that the only client side configurations required are: + +* Security protocol definition +* Yarn Resource Manager endpoints +* Storage service (HDFS or Cloud storage). + +Once the Hadoop client configuration is defined, it is only a matter of pointing the DQ Agent at the folder that contains the client configuration files. The DQ Agent is then able to use the Hadoop client configuration to submit jobs to the specified Hadoop cluster. + +{% hint style="info" %} +DQ jobs running on Hadoop are Spark jobs. DQ will use the storage platform defined in the "fs.defaultFS" setting to distribute all of the required Spark libraries and specified dependency packages like drivers files. This allows DQ to use a version of Spark that is different than the one provided by the cluster. If it is a requirement to use the Spark version provided by the target Hadoop cluster, obtain and use a copy of the yarn-site.xml and core-site.xml from the cluster. +{% endhint %} + +### Create Config Folder + +``` +cd $OWL_HOME +mkdir -p config/hadoop +echo "export HADOOP_CONF_DIR=$OWL_HOME/config/hadoop" >> config/owl-env.sh +bin/owlmanage.sh restart=owlagent +``` + +### Minimum Config (Kerberos Disabled, TLS Disabled) + +This configuration would typical only be applicable in Cloud Hadoop scenarios (EMR/Dataproc/HDI). Cloud Hadoop clusters are ephemeral and do not store any data as the data is stored in and is secured by Cloud Storage. + +``` +export RESOURCE_MANAGER= +export NAME_NODE= + +echo " + + + hadoop.security.authentication + simple + + + hadoop.rpc.protection + authentication + + + fs.defaultFS + hdfs://$NAME_NODE:8020 + + +" >> $OWL_HOME/config/hadoop/core-site.xml + +echo " + + + yarn.resourcemanager.scheduler.address + $RESOURCE_MANAGER:8030 + + + yarn.resourcemanager.address + $RESOURCE_MANAGER:8032 + + + yarn.resourcemanager.webapp.address + $RESOURCE_MANAGER:8088 + + +" >> $OWL_HOME/config/hadoop/yarn-site.xml +``` + +{% hint style="info" %} +When deploying a Cloud Service Hadoop cluster from any of the major Cloud platforms, it is possible to use Cloud Storage rather than HDFS for dependency package staging and distribution. To achieve this, create a new storage bucket and ensure that both the Hadoop cluster and the instance running DQ Agent have access to it. This is usually accomplished using a Role that is attached to the infrastructure. For example, AWS Instance Role with bucket access policies. Then, set "fs.defaultFS" in core-site.xml to the bucket path instead of HDFS. +{% endhint %} + +Once the Hadoop client configuration has been created, navigate to Agent Management console from the Admin Console and configure the agent to use Yarn (Hadoop resource scheduler) as the Default Master and set the Default Deployment Mode to "Cluster". + +![](<../../.gitbook/assets/Screenshot 2021-06-21 at 8.42.19 AM.png>) + +### Kerberos Secured with Resource Manager TLS enabled + +Typically, Hadoop cluster that are deployed on-premises are multi-tenant and not ephemeral. This means they must be secured using Kerberos. In addition, all endpoints with HTTP endpoints will have TLS enabled. In addition HDFS may be configured for a more secure communication using additional RPC encryption. + +``` +export RESOURCE_MANAGER= +export NAME_NODE= +export KERBEROS_DOMAIN= +export HDFS_RPC_PROTECTION= + +echo " + + + hadoop.security.authentication + kerberos + + + hadoop.rpc.protection + $HDFS_RPC_PROTECTION + + + fs.defaultFS + hdfs://$NAME_NODE:8020 + + +" >> $OWL_HOME/config/hadoop/core-site.xml + +echo " + + + hadoop.security.authentication + HDFS/_HOST@$KERBEROS_DOMAIN + + +" >> $OWL_HOME/config/hadoop/hdfs-site.xml + +echo " + + + yarn.resourcemanager.scheduler.address + $RESOURCE_MANAGER:8030 + + + yarn.resourcemanager.address + $RESOURCE_MANAGER:8032 + + + yarn.resourcemanager.webapp.https.address + $RESOURCE_MANAGER:8090 + + +" >> $OWL_HOME/config/hadoop/yarn-site.xml +``` + +When the target Hadoop cluster is secured by Kerberos, DQ checks require a Kerberos credential. This typically means that the DQ Agent will need to be configured to include a Kerberos keytab with each DQ check. Access the DQ Agent configuration page from the Admin Console and configure the "Freeform Append" setting with the -sparkprinc \ -sparkkeytab \. + +![](<../../.gitbook/assets/Screenshot 2021-06-21 at 8.38.56 AM.png>) diff --git a/installation/standalone/README.md b/installation/standalone/README.md new file mode 100644 index 00000000..a7d80166 --- /dev/null +++ b/installation/standalone/README.md @@ -0,0 +1,5 @@ +# Standalone + +When large scale and high concurrency checks are not required, DQ can be installed and operated entirely on a single host. In this mode, DQ will leverage a **Spark Standalone pseudo cluster** where the master and workers run and use resources from the same server. DQ also requires a Postgres database for storage and Java 8 for running the DQ web application. It is possible to install each of the Spark, Postgres, and Java 8 components separately and install DQ on top of existing components. However, we offer a full installation package that installs these components in off-line mode and install DQ in one server. + +![Fig 1: Architecture overview of Full Standalone Installation mode](<../../.gitbook/assets/Screenshot 2021-06-14 at 5.44.53 PM.png>) diff --git a/installation/standalone/full-install.md b/installation/standalone/full-install.md new file mode 100644 index 00000000..5a4514d4 --- /dev/null +++ b/installation/standalone/full-install.md @@ -0,0 +1,533 @@ +# Standalone Install + +## Step 0. Setup Tutorial Assumptions + +We assume that a server running Centos 7 or RHEL 7 is setup and ready to install DQ in the home directory (base path: `OWL_BASE`) under subdirectory `owl`(install path: `$OWL_BASE/owl`). There is no requirement for DQ to be installed in the home directory, but the DQ Full Installation script may lead to permission-denied issue during local Postgres server installation if paths other than home directory is used. If so, please adjust your directory permission to allow the installation script a write access to the Postgres data folder. + +This tutorial assumes that you are installing DQ on a brand new compute instance on Google Cloud Platform. Google Cloud SDK setup with proper user permission is assumed. This is optional, as you are free to create Full Standalone Installation setup on any cloud service provider or on-premise. + +Please refer to the _**GOAL**_** ** paragraph for the intended outcome of each step and modify accordingly + +{% hint style="info" %} +The full install package supports Centos 7 and RHEL 7. If another OS flavor is required, please follow the basic install process. +{% endhint %} + +``` +# Create new GCP Compute Instance named "install" +gcloud compute instances create install \ + --image=centos-7-v20210701 \ + --image-project=centos-cloud \ + --machine-type=e2-standard-4 + +# SSH into the instance as user "centos" +gcloud compute ssh --zone "us-central1-a" --project "gcp-example-project" "centos@full-standalone-installation" +``` + +{% hint style="success" %} +**GOAL** + +1. Create a new compute instance on a cloud provider (if applicable) +2. Access the server where DQ will be installed. +{% endhint %} + +## Step 1. Download DQ Full Package + +Download full package tarball using the signed link to the full package tarball provided by the DQ Team. Replace `` with the link provided. + +``` +### Go to the OWL_BASE (home directory of the user is most common) +### This example we will use /home/owldq installing as the user owldq + +cd /home/owldq + +### Download & untar +curl -o dq-full-package.tar.gz "" +tar -xvf dq-full-package.tar.gz + +### Clean-up unnecessary tarball (optional) +rm dq-full-package.tar.gz +``` + +{% hint style="success" %} +**GOAL** + +1. Download the full package tarball and place it in the `$OWL_BASE` (home directory). Download via `curl` or upload directly via FTP. The tarball name is assumed to be `dq-full-package.tar.gz`for sake of simplicity. +2. Untar`dq-full-package.tar.gz` to `OWL_BASE`. +{% endhint %} + +## Step 2. Install DQ + Postgres + Spark + +First set some variables for `OWL_BASE` (where to install DQ. In this tutorial, you are already in the directory that you want to install), `OWL_METASTORE_USER` (the Postgres username used by DQ Web Application to access Postgres storage), and `OWL_METASTORE_PASS` (the Postgres password used by DQ Web Application to access Postgres storage). + +``` +### base path that you want owl installed. No trailing + +export OWL_BASE=$(pwd) +export OWL_METASTORE_USER=postgres +# minimum complexity recommended (18 length, upper, lower, number, symbol) +# example below +export OWL_METASTORE_PASS=H55Mt5EbXh1a%$aiX6 +``` + +`dq-package-full.tar.gz` that you untarred contains installation packages for Java 8, Postgres 11, and Spark. There is no need to download these components. These off-line installation components are located in `$(pwd)/package/install-packages` . + +![List of off-line installed packages](../../.gitbook/assets/full-package-install-packages.png) + +One of the files extracted from the tarball is `setup.sh`. This script installs DQ and the required components. If a component already exist (e.g. Java 8 is already installed and `$JAVA_HOME` is set), then that component is not installed (i.e. Java 8 installation is skipped). + +To control which components are installed, use `-options=...`parameter. The argument provided should be comma-delimited list of components to install (valid inputs: `spark`, `postgres`, `owlweb`, and `owlagent`. `-options=postgres,spark,owlweb,owlagent` means "install Postgres, Spark pseudo cluster, Owl Web Application, and Owl Agent". Note that Java is not part of the options. Java 8 installation is automatically checked and installed/skipped depending on availability. + +You must at minimum specify `-options=spark,owlweb,owlagent` if you independently installed Postgres or using an external Postgres connection (as you can see in Step #3 if you choose that installation route) + +``` +### The following installs PostgresDB locally as part of OwlDQ install + +./setup.sh \ + -owlbase=$OWL_BASE \ + -user=$OWL_METASTORE_USER \ + -pgpassword=$OWL_METASTORE_PASS \ + -options=postgres,spark,owlweb,owlagent +``` + +{% hint style="info" %} +If prompted to install Java 8 because you do not have Java 8 installed, accept to install from local package. +{% endhint %} + +{% hint style="info" %} +You will be prompted for where to install Postgres like the following image: + +``` +Postgres DB needs to be intialized. Default location = /postgres/data +to change path please enter a FULL valid path for Postgres and hit +DB Path [ /owl/postgres/data ] = +``` + +If the data files for the Postgres database need to be hosted at a specific location, provide it during this prompt. **Make sure the directory is writable**. Otherwise, just press \ to install the data files into `$OWL_BASE/owl/postgres/data`. The default suggested path does not have permission issue if you chose home directory as `OWL_BASE` +{% endhint %} + +If no exceptions occurred and installation was successful, then the process will complete with the following output. + +``` +installing owlweb +starting owlweb +starting owl-web +installing agent +not starting agent +install complete +please use owl owlmanage utility to configure license key and start owl-agent after owl-web successfully starts up +``` + +{% hint style="success" %} +**GOAL** + +1. Specify `OWL_BASE` path where DQ will be installed and specify Postgres environment variables +2. Install DQ Web with Postgres and Spark linked to DQ Agent (**all files will be in `$OWL_BASE/owl` sub-directory**) using `setup.sh` script provided. \ + The location of `OWL_BASE` and Postgres are configurable, but we advise you to take the defaults. +{% endhint %} + +## Step 3_._ Install DQ + Spark and use existing Postgres (advanced) + +{% hint style="warning" %} +Skip Step 3 if you opted to install Postgres and performed Step 2 instead. +{% endhint %} + +{% hint style="warning" %} +We only recommend Step 3 over Step 2 for advanced DQ Installer +{% endhint %} + +If you have already installed DQ from the previous step, then skip this step. This is only for those who want to use external Postgres (e.g. use GCP Cloud SQL service as the Postgres metadata storage). If you have an existing Postgres installation, then everything in the previous step applies except the Postgres data path prompt and the `setup.sh` command + +Refer to the Step #2 for details on what `OWL_BASE`, `OWL_METASTORE_USER` , and `OWL_METASTORE_PASS` are. + +``` +# base path that you want owl installed. No trailing + +export OWL_BASE=$(pwd) +export OWL_METASTORE_USER=postgres +# minimum complexity recommended (18 length, upper, lower, number, symbol) +# example below +export OWL_METASTORE_PASS=H55Mt5EbXh1a%$aiX6 +``` + +Run the following installation script. Note the missing "postgres" in `-options` and new parameter `-pgserver`. This `-pgserver` could point to any URL that the standalone instance has access to. + +``` +# The following does not install PostgresDB and +# uses existing PostgresDB server located in localhost:5432 with "postgres" database +./setup.sh \ + -owlbase=$OWL_BASE \ + -user=$OWL_METASTORE_USER \ + -pgpassword=$OWL_METASTORE_PASS \ + -options=spark,owlweb,owlagent \ + -pgserver="localhost:5432/postgres" +``` + +The database named `postgres` is used by default as DQ metadata storage. Changing this database name is out-of-scope for Full Standalone Installation. Contact DQ Team for assistance. + +{% hint style="success" %} +**GOAL** + +1. Specify `OWL_BASE` path where DQ will be installed and specify Postgres environment variables +2. Install DQ Web and Spark linked to DQ Agent (**all files will be in `$OWL_BASE/owl` sub-directory**) using `setup.sh` script provided and link DQ Web to an existing Postgres server. +{% endhint %} + +## Step 4. Verify DQ and Spark Installation + +The installation process will start the DQ Web Application. This process will handle initializing the Postgres metadata storage schema in Postgres (under the database named `postgres`). This process must complete successfully before the DQ Agent can be started. Wait approximately 1 minute for the Postgres metadata storage schema to be populated. If you can access DQ Web using `:9000` using a Web browser, then this means you have successfully installed DQ. + +![Fig 2: Login page of DQ Web UI](../../.gitbook/assets/DQ-Login.png) + +Next, verify that the Spark Cluster has started and is available to run DQ checks using `:`8080 Take note of the Spark Master url (starting with `spark://...`). This will be required during DQ Agent configuration. + +![Fig 3: Spark Master Web UI](<../../.gitbook/assets/Screenshot 2021-06-14 at 4.17.30 PM.png>) + +## Step 5. Set License Key + +In order for DQ to run checks on data, the DQ Agent must be configured with a license key. Replace `` with a valid license key provided by Collibra. + +``` +cd $OWL_BASE/owl/bin +./owlmanage.sh setlic= + +# expected output: +# > License Accepted new date: +``` + +## Step 6. Set DQ Agent Configuration + +Next, start the DQ Agent process to enable processing of DQ checks. + +``` +cd $OWL_BASE/owl/bin +./owlmanage.sh start=owlagent + +# Verify "agent.properties" file is created +cd $OWL_BASE/owl/config +``` + +When the script successfully runs, `$OWL_BASE/owl/config` folder will contain a file called `agent.properties`. This file contains agent id # of agents installed in this machine. Since this is the first non-default agent installed, the expected agent id is 2. Verify `agent.properties` file is created. Your`agent.properties` is expected to have different timestamp, but you should see `agentid=2` + +``` +cd $OWL_BASE/owl/config +cat agent.properties + +# expected output: +> #Tue Jul 13 22:26:19 UTC 2021 +> agentid=2 +``` + +Once the DQ Agent starts, it needs to be configured in DQ Web in order to successfully submit jobs to the local Spark (pseudo) cluster. + +The new agent has been setup with the template base path `/opt` and install path `/opt/owl`. The `owlmanage.sh start=owlagent` script does not respect `OWL_BASE` environment. **We need to edit the Agent Configuration to follow our** `OWL_BASE` + +Follow the steps on [How To Configure Agent via UI](https://docs.owl-analytics.com/installation/agent-configuration#how-to-configure-agent-via-ui) page to configure the newly created DQ Agent and edit the following parameters in DQ Agent #2. + +* Replace all occurrence of `/opt/owl` with your `$OWL_BASE/owl/`in **Base Path**, **Collibra DQ Core JAR**, **Collibra DQ Core Logs**, **Collibra DQ Script**, and **Collibra DQ Web Logs**. + * Note that **Base Path** here does not refer to `OWL_BASE` +* Replace **Default Master** value with the Spark URL from Fig 3 +* Replace **Number of Executors(s)**, **Executor Memory (GB)**, **Driver Memory (GB)** to a reasonable default (depending on how large your instance is) + +Refer to [Agent Configuration Parameters](https://docs.owl-analytics.com/installation/agent-configuration#agent-configuration-parameters) for parameters descriptions. + +![Fig 4: Expected final output of edited agent based on this tutorial](<../../.gitbook/assets/image (67).png>) + +## Step 7. Create DB Connection for DQ Job + +Follow the steps on [How to Add DB Connection via UI](https://docs.owl-analytics.com/installation/agent-configuration#how-to-add-db-connection-via-ui) page to add `metastore` database connection. For demo purposes, we will run a DQ Job against local DQ Metadata Storage. + +Follow the steps on [How To Link DB Connection to Agent via UI](https://docs.owl-analytics.com/installation/agent-configuration#how-to-link-db-connection-to-agent-via-ui) page to configure newly created DQ Agent. + +Click the compass icon in the navigation pane to navigate to the Explorer Page. Click on the "metastore" connection, select the "public" schema, and then select the first table in the resulting list of tables. Once the preview and scope tab comes up, click "Build Model". When the Profile page comes up, click the "Run" button. + +![](<../../.gitbook/assets/Screenshot 2021-06-14 at 5.03.40 PM.png>) + +On the Run page, click the "Estimate Job" button, acknowledge the resource recommendations, and then click the "Run" button. + +![](<../../.gitbook/assets/Screenshot 2021-06-14 at 5.05.31 PM.png>) + +Click the clock icon in the navigation pane to navigate to the Jobs Page. Wait 10 seconds and then click refresh several times with a few seconds in between clicks. The test DQ check should show and progress through a sequence of activities before settling in "Finished "status. + +![](<../../.gitbook/assets/Screenshot 2021-06-14 at 5.27.04 PM.png>) + +## Generally Available Build Versions + +* Default Build = Spark 2.3.0 +* Spark 2.4.5 +* Spark 3.0.1 +* HDP 3 +* CDH5 +* CDH6-NOLOG +* K8s + +### Helpful Commands + +``` +### Setting permissions on your pem file for ssh access + +chmod 400 ~/Downloads/ssh_pem_key +``` + +``` +### Postgres data directly initialization failed +### Postgres permission denied errors +### sed: can't read /home/owldq/owl/postgres/data/postgresql.conf: Permission denied + +sudo rm -rf /home/owldq/owl/postgres +chmod -R 755 /home/owldq + +### Reinstall just postgres +./setup.sh -owlbase=$OWL_BASE -user=$OWL_METASTORE_USER -pgpassword=$OWL_METASTORE_PASS -options=postgres +``` + +``` +### Spark standalone permission denied after using ./start-all.sh + +ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa +cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys +``` + +``` +### Changing permissiongs on individual log files + +sudo chmod 777 /home/owldq/owl/pids/owl-agent.pid +sudo chmod 777 /home/owldq/owl/pids/owl-web.pid +``` + +``` +### Getting the hostname of the instance + +hostname -f +``` + +``` +### Checking worker nodes disk space + +sudo du -ah | sort -hr | head -5 +sudo find /home/owldq/owl/spark/work/* -mtime +1 -type f -delete +``` + +``` +### Redirect Spark scratch to another location +SPARK_LOCAL_DIRS=/mnt/disks/sdb/tmp + +### Set Spark to delete older files +export SPARK_WORKER_OPTS="${SPARK_WORKER_OPTS} -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=1800 -Dspark.worker.cleanup.appDataTtl=3600" + +``` + +Or change Spark storage with an agent configuration `-conf spark.local.dir=/home/owldq/owl/owltmp` + +![](<../../.gitbook/assets/image (106).png>) + +## Troubleshooting Tip: Add Spark Home Environment Variables to Profile + +``` +### Adding ENV variables to bash profile + +### Variable 'owldq' below should be updated wherever installed e.g. centos + +vi ~/.bash_profile +export SPARK_HOME=/home/owldq/owl/spark +export PATH=$SPARK_HOME/bin:$PATH + +### Add to owl-env.sh for standalone install + +vi /home/owldq/owl/config/owl-env.sh +export SPARK_HOME=/home/owldq/owl/spark +export PATH=$SPARK_HOME/bin:$PATH +``` + +## Check Processes are Running + +``` +### Checking PIDS for different components + +ps -aef|grep postgres +ps -aef|grep owl-web +ps -aef|grep owl-agent +ps -aef|grep spark + +``` + +## Starting Components + +``` +### Restart different components + +cd /home/owldq/owl/bin/ +./owlmanage.sh start=postgres +./owlmanage.sh start=owlagent +./owlmanage.sh start=owlweb + +cd /home/owldq/owl/spark/sbin/ +./stop-all.sh +./start-all.sh +``` + +## Configuration Options + +### Setup.sh arguments + +**-non-interactive** skip asking to accept JAVA license agreement + +**-skipSpark** skips the extraction of spark components + +**-stop** do not automatically start up all components (orient,owl-web,zeppelin,postgres) + +**-port=** set owlweb application to use defined port + +**-user=** Optional Parameter (default will be current user) set the user to run owl as. + +\-**owlbase=** set base path to where you want owl installed + +**-owlpackage=** Optional Parameter (default is current working directory) set owl package directory + +**-help** display this help and exit + +**-options=** the different owl components to install (comma separated list) --- owlagent,owlweb,zeppelin,postgres,orient,spark + +**-pgpassword=** password to use to set for the postgres metastore (unattended install) + +**-pgserver=** name of the postgres server example = owl-postgres-host.example.com:5432/owldb (unattended install) + +**-opassword=** password for the orient graph DB (unattended install) + +#### Example: + +* The Owl tar ball has been extracted to this folder on my EC2 Instance: _**/home/ec2-user/packages/**_ +* Owl will be running as the _**ec2-user**_ +* The owl-web application will run on port _**9000**_ +* The base location for the setup.sh script to create the owl folder and place all content under owl will be: _**/home/ec2-user/**_ + +_**./setup.sh -port=9000 -user=ec2-user -owlbase=/home/ec2-user -owlpackage=/home/ec2-user/package**_ + +#### Example installing just the agent (perhaps on an Edge node of a hadoop cluster): + +* The Package has been extracted to this folder on my EC2 Instance: _**/home/ec2-user/packages/**_ +* Owl-agent will be running as the _**ec2-user**_ +* The base location for the setup.sh script to create the owl folder and place all packages under owl will be: _**/home/ec2-user/**_ + +_**./setup.sh -user=ec2-user -owlbase=/home/ec2-user -owlpackage=/home/ec2-user/package -options=owlagent**_ + +{% hint style="info" %} +When installing different features questions will be asked + +* postgres = Postgres DBPassword needs to be supplied +* orient = Orient DBPassword needs to be supplied +* If postgres is not being installed (such as agent install only) postgres metastore server name needs to be supplied +{% endhint %} + +### Launching and Administering owl: + +When the setup.sh script finishes by default software is automatically started. The setup.sh also creates the owlmanage.sh script which allows for stopping and starting of all owl services or some components of services.The setup script will also generate an owl-env.sh script that will hold the main variables that are reused across components (see owl-env.sh under the config directory). + +### Owl Directory Structure after running Setup.sh + +export ORIENTDB\_HOST="localhost" + +![](https://files.gitbook.com/v0/b/gitbook-28427.appspot.com/o/assets%2F-L\_xJcI5Uc0S1x9JC6xQ%2F-LiQA1uhYoTELY58hjz\_%2F-LiQBsDxBzzkDzLkf-DB%2Fimage.png?alt=media\&token=05fb9d40-7697-4b09-b70b-f481ca0f292a) + +#### Configuration / ENV settings within owl-env.sh + +Contents of the Owl-env.sh script and what is is used for. + +| OWL-ENV.SH Scripts | Meaning | +| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| export ORIENTDB\_PORT="2424" | Port that OrientDB is using | +| export SPARK\_CONF\_DIR="/home/danielrice/owl/cdh-spark-conf" | directory on machine where the spark conf directory resides. | +| export INSTALL\_PATH="/home/danielrice/owl" | Installation directory of Owl | +| export JAVA\_HOME="/home/danielrice/jdk1.8.0\_131" | Java Home for Owl to leverage | +| export LOG\_PATH="/home/danielrice/owl/log" | Log path | +| export BASE\_PATH="/home/danielrice" | The base location under which the owl director resides | +| export SPARK\_MAJOR\_VERSION=2 | Spark Major version. Owl only supports 2+ version of spark | +| export OWL\_LIBS="/home/danielrice/owl/libs" | Lib Directory to inject in spark -submit jobs | +| export USE\_LIBS=0 #1 is on, 0 is off | use the lib directory or not. 0 is the default. | +| export SPARKSUBMITEXE="spark-submit" | Spark submit executable command. CDH using spark2-submit as example | +| export ext\_pass\_manage=0 #0 to disable 1 to enable | If using a password management system. You can enable for password to be pulled from it. | +| export ext\_pass\_script="/opt/owl/bin/getpassword.sh" | Leverage password script to execute a get password script from the vault. | +| TIMEOUT=900 #15 minutes in seconds | Owl-Web user time out limits | +| PORT=9003 #owl-web port NUMBER | default port to use for owl-web | +| SPRING\_LOG\_LEVEL=ERROR | logging level to be displayed in the owl-web.log | +| SPRING\_DATASOURCE\_DRIVER\_CLASS\_NAME=org.postgresql.Driver | driver class name for postgres metastore (used by web) | +| export SPRING\_DATASOURCE\_URL=jdbc:postgresql://localhost:5432/postgres | JDBC connection string to Owl Postgres metastore | +| export SPRING\_DATASOURCE\_USERNAME=danielrice | Owl Postgres username | +| export SPRING\_DATASOURCE\_PASSWORD=3+017wfY1l1vmsvGYAyUcw5zGL | Owl Postgres password | +| export AUTOCLEAN=TRUE/FALSE | TRUE/FALSE Enable/Disable automatically delete old datasets | +| export DATASETS\_PER\_ROW=200000 | Delete datasets after this threshold is hit (must be greater than the default to change) | +| export ROW\_COUNT\_THRESHOLD=300000 | Delete rows after this threshold is hit (must be greater than the default to change) | +| export SERVER\_HTTP\_ENABLED=true | Enabling HTTP to owl web | +| export OWL\_ENC=OFF #JAVA for java encryption | Enable Encryption (NOTE need to add to owl.properties also). Has to be in form owl.enc=OFF within owl.properties file to disable, and in this form owl.enc=JAVA to enable. the owl.properties file is located in the owl install path /config folder (/opt/owl/config). | +| PGDBPATH=/home/danielrice/owl/owl-postgres/bin/data | Path for Postgres DB | +| export RUNS\_THRESHOLD=5000 | Delete runs after this threshold is hit (must be greater than the default to change) | +| export HTTP\_SECONDARY\_PORT=9001 | Secondary HTTP port to use which is useful when SSL is enabled | +| export SERVER\_PORT=9000 | same as PORT | +| export SERVER\_HTTPS\_ENABLED=true | enabling of SSL | +| export SERVER\_SSL\_KEY\_TYPE=PKCS12 | certificate trust store | +| export SERVER\_SSL\_KEY\_PASS=t2lMFWEHsQha3QaWnNaR8ALaFPH15Mh9 | certificate key password | +| export SERVER\_SSL\_KEY\_ALIAS=owl | certificate key alias | +| export SERVER\_REQUIRE\_SSL=true | Override HTTP on and force HTTPS regardless of HTTP settings | +| export MULTITENANTMODE=FALSE | Flipping to TRUE will enable multi tenant support | +| export multiTenantSchemaHub=owlhub | schema name used for multi tenancy | +| export OWL\_SPARKLOG\_ENABLE=false | Enabling deeper spark logs when toggled to true | + +#### Configuration / owl.properties file + +| Example | Meaning | +| ----------------------------------------------------------------- | ----------------------------------------------------- | +| key=XXXXXX | The license key | +| spring.datasource.url=jdbc:postgresql://localhost:5432/postgres | connection string to owl metastore (used by owl-core) | +| spring.datasource.password=xxxxxx | password to owl metastore (used by owl-core) | +| spring.datasource.username=xxxxxx | username to owl metastore (used by owl-core) | +| spring.datasource.driver-class-name=com.owl.org.postgresql.Driver | shaded postgres driver class name | +| orientdb.user=root | orient username | +| orientdb.pass=xxxxx | orient password | +| **spring.agent.datasource.url** | **jdbc:postgresql://$host:$port/owltrunk** | +| **spring.agent.datasource.username** | **{user}** | +| **spring.agent.datasource.password** | **{password}** | +| **spring.agent.datasource.driver-class-name** | **org.postgresql.Driver** | + + + +## Starting Spark + +{% embed url="https://spark.apache.org/docs/latest/spark-standalone.html#cluster-launch-scripts" %} + +### Launch Scripts + +To launch a Spark standalone cluster with the launch scripts, you should create a file called conf/workers in your Spark directory, which must contain the hostnames of all the machines where you intend to start Spark workers, one per line. If conf/workers does not exist, the launch scripts defaults to a single machine (localhost), which is useful for testing. Note, the master machine accesses each of the worker machines via ssh. By default, ssh is run in parallel and requires password-less (using a private key) access to be setup. If you do not have a password-less setup, you can set the environment variable SPARK\_SSH\_FOREGROUND and serially provide a password for each worker. + +Once you’ve set up this file, you can launch or stop your cluster with the following shell scripts, based on Hadoop’s deploy scripts, and available in `SPARK_HOME/sbin`: + +* `sbin/start-master.sh` - Starts a master instance on the machine the script is executed on. +* `sbin/start-workers.sh` - Starts a worker instance on each machine specified in the `conf/workers` file. +* `sbin/start-worker.sh` - Starts a worker instance on the machine the script is executed on. +* `sbin/start-all.sh` - Starts both a master and a number of workers as described above. +* `sbin/stop-master.sh` - Stops the master that was started via the `sbin/start-master.sh` script. +* `sbin/stop-worker.sh` - Stops all worker instances on the machine the script is executed on. +* `sbin/stop-workers.sh` - Stops all worker instances on the machines specified in the `conf/workers` file. +* `sbin/stop-all.sh` - Stops both the master and the workers as described above. + +Note that these scripts must be executed on the machine you want to run the Spark master on, not your local machine. + +``` +### Starting Spark Standalone + +cd /home/owldq/owl/spark/sbin +./start-all.sh + +### Stopping Spark +cd /home/owldq/owl/spark/sbin +./stop-all.sh +``` + +``` +### Starting Spark with Separate Workers + +SPARK_WORKER_OPTS=" -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=1799 -Dspark.worker.cleanup.appDataTtl=3600" + +### 1 start master +/home/owldq/owl/spark/sbin/start-master.sh + +### 2 start workers +SPARK_WORKER_INSTANCES=3;/home/owldq/owl/spark/sbin/start-slave.sh spark://$(hostname):7077 -c 5 -m 20g +``` diff --git a/installation/standalone/standalone-sizing.md b/installation/standalone/standalone-sizing.md new file mode 100644 index 00000000..a54794e6 --- /dev/null +++ b/installation/standalone/standalone-sizing.md @@ -0,0 +1,53 @@ +# Standalone Sizing + +## Small Tier - 16 Core, 128G RAM (r5.4xlarge / E16s v3) + +| Component | RAM | Cores | +| --------- | ---- | ----- | +| Web | 2g | 2 | +| Postgres | 2g | 2 | +| Spark | 100g | 10 | +| Overhead | 10g | 2 | + +## Medium Tier - 32 Core, 256G RAM (r5.8xlarge / E32s v3) + +| Component | RAM | Cores | +| --------- | ---- | ----- | +| Web | 2g | 2 | +| Postgres | 2g | 2 | +| Spark | 250g | 26 | +| Overhead | 10g | 2 | + +## Large Tier - 64 Core, 512G RAM (r5.16xlarge / E64s v3) + +| Component | RAM | Cores | +| --------- | ---- | ----- | +| Web | 4g | 3 | +| Postgres | 4g | 3 | +| Spark | 486g | 54 | +| Overhead | 18g | 4 | + +## Estimates + +Sizing should allow headroom and based on peak concurrency and peak volume requirements. If concurrency is not a requirement, you just need to size for peak volume (largest tables). Best practice to efficiently scan is to scope the job by selecting critical columns. See [performance tuning](../../benchmarks/performance-tuning/) for more information. + +| Bytes per Cell | Rows | Columns | Gigabytes | Gigabytes for Spark (3x) | +| -------------- | ---------------- | ------- | --------- | ------------------------ | +| 16 | 1,000,000.00 | 25 | 0.4 | 1.2 | +| 16 | 10,000,000.00 | 25 | 4 | 12 | +| 16 | 100,000,000.00 | 25 | 40 | 120 | +| 16 | 1,000,000.00 | 50 | 0.8 | 2.4 | +| 16 | 10,000,000.00 | 50 | 8 | 24 | +| 16 | 100,000,000.00 | 50 | 80 | 240 | +| 16 | 1,000,000.00 | 100 | 1.6 | 4.8 | +| 16 | 10,000,000.00 | 100 | 16 | 48 | +| 16 | 1,000,000,000.00 | 100 | 1600 | 4800 | +| 16 | 100,000,000.00 | 100 | 160 | 480 | +| 16 | 1,000,000.00 | 200 | 3.2 | 9.6 | +| 16 | 10,000,000.00 | 200 | 32 | 96 | +| 16 | 100,000,000.00 | 200 | 320 | 960 | +| 16 | 1,000,000,000.00 | 200 | 3200 | 9600 | + +## Cluster + +If your program requires more horsepower or (spark) workers than the example tiers above which is fairly common in Fortune 500 companies than you should consider the horizontal and ephemeral scale of a cluster. Common examples are Amazon EMR, Cloudera CDP etc... OwlDQ is built to scale up horizontally and can scale to hundreds of nodes. diff --git a/installation/standalone/standalone-upgrade.md b/installation/standalone/standalone-upgrade.md new file mode 100644 index 00000000..d89de89b --- /dev/null +++ b/installation/standalone/standalone-upgrade.md @@ -0,0 +1,90 @@ +# Standalone Upgrade + +{% hint style="success" %} +Before proceeding with any upgrades, please remember to backup your DQ Metastore +{% endhint %} + +{% hint style="danger" %} +Please remember that rolling back Collibra DQ to a prior version is not supported. Please contact Collibra Support with any questions. +{% endhint %} + +### Download DQ Upgrade Package + +**Note: Beginning December 2021, all Collibra DQ customers upgrading or patching will be receiving the Full package (vs. Base) and should follow the same Upgrade steps below** + +Download tarball using the signed link to the full package tarball provided by Collibra. Replace `` with the link provided. + +``` +### Go to the OWL_BASE (home directory of the user is most common) +### This example we will use /home/owldq installing as the user owldq + +cd /home/owldq + +### Download & untar +curl -o dq-full-package.tar.gz "" +tar -xvf dq-full-package.tar.gz + +### Clean-up unnecessary tarball (optional) +rm dq-full-package.tar.gz +``` + +### Upgrade Steps + +1. Copy the contents of the provided package e.g. owl-\-\-package-full.tar.gz to the system being upgraded (extract contents) + * Best practice: Untar contents into a uniquely named folder e.g. 2021-12-dq-upgrade +2. Stop owlweb using `./owlmanage.sh stop=owlweb` +3. Stop owlagent using `./owlmanage.sh stop=owlagent` +4. Move old jars from `owl/bin` + 1. `mv owl-webapp--.jar /tmp` + 2. `mv owl-agent--.jar /tmp` + 3. `mv owl-core--.jar /tmp` +5. Copy new jars into the `owl/bin` folder from the extracted package + 1. `mv owl-webapp--.jar /home/owldq/owl/bin` + 2. `mv owl-agent--.jar /home/owldq/owl/bin` + 3. `mv owl-core--.jar /home/owldq/owl/bin` +6. run `./owlmanage.sh start=owlweb` to start the owl-web application +7. run `./owlmanage.sh start=owlagent` to start owlagent + +### Additional Notes / Steps Due To Log4J (December 2021) + +#### Additional Step 1: Place Log4j-1.2-api-2.17.1.jar (as of 2022.02) into /\/owl/spark/jars + +* #### Note: Was Log4j-1.2-api-2.17.0.jar in 2021.12 and 2022.01 + +**Who: All Collibra DQ customers, particularly those leveraging CLI mode** + +1. Navigate to the same folder where the Collibra provided upgrade package was extracted +2. Navigate to \/packages/install-packages +3. Extract the needed log4j-1.2-api-2.17.1.jar via the command:`tar -xvf spark-extras.tar.gz spark-extras/log4j-1.2-api-2.17.1.jar` +4. Move the log4j-1.2-api-2.17.1.jar file into /\/spark/jars folder + +**FAQ** + +Q: (When) do I need to move Log4j-1.2-api-2.17.1.jar before or after swapping the main Collibra DQ jars? + +* A: Sequence does not matter. + +Q: (What) if I don't follow these additional upgrade steps? + +* A: If your `SPARK_SUBMIT_MODE` within owl-env.sh is set to `SPARK_SUBMIT_MODE=native`, Collibra DQ will function properly without the above additional upgrade step, with the exception of CLI mode + +#### Additional Step 2: Remove a legacy properties file + +**Who: Only Collibra DQ customers upgrading Agents installed on Cloudera CDP Hadoop Edge Nodes** + +1. Navigate to /\/owl/config/ +2. Remove `log4j-cluster.properties` file + +**FAQ** + +Q: (When) do I need to remove log4j-cluster.properties before or after swapping the main Collibra DQ jars? + +* A: Remove the file before restarting owl-agent. Otherwise, stop owl-agent again, remove the file, then restart owl-agent. + +Q: (What) if I don't follow these additional steps? + +* A: For customers using agents on Hadoop edge nodes, they will receive errors when running DQ Jobs due to engaging a method that no longer exists. + +Q: What should I do if I am not a vendor-supported Cloudera CDP version? + +* A: Our testing and guidance mainly applies to vendor-supported (non-EOL) Cloudera CDP versions. Other Hadoop variants may handle logging differently and may require the legacy properties file. In short, feel free to first upgrade without this step, then remove the log4j-cluster.properties file if DQ Jobs are running into issues. diff --git a/integration/dq-connector.md b/integration/dq-connector.md new file mode 100644 index 00000000..70b6c7d5 --- /dev/null +++ b/integration/dq-connector.md @@ -0,0 +1,243 @@ +# DQ Connector + +## Current Status: \[Tech Preview] + +## Benefits + +The Native DQ Connector brings intelligence from **Collibra Data Quality** into **Collibra Data Intelligence Cloud**. Once this integration is established, you will be able to bring in your Data Quality user-defined rules, metrics, and dimensions into **Collibra Data Catalog**. + +_Please note: Only data sources ingested by both Collibra Data Catalog and Collibra Data Quality will be able to synchronize Data Quality assets_ + +## Step 0: Prerequisites + +| **Resource** | Notes | +| -------------------------------- | --------------------------------------------------- | +| Collibra Edge Site | DQ Connector is a capability of Edge | +| Collibra Data Intelligence Cloud | 2021.07 Release (or newer) | +| Collibra Data Quality | 2.15 (or newer) | +| Database(s) and Driver(s) | Proper Access and Credentials (Username / Password) | + +{% hint style="success" %} +Let's proceed after gathering all prerequisites! +{% endhint %} + +## Step 1: Create and Configure Edge and DQ Connector + +**1A. Create Edge site and Add Name e.g. 'Collibra-DQ-Edge' and Description (One-Time)** + +![Where: Collibra Data Intelligence Cloud Settings -\> Edge -\> Click ‘Create Edge site’](https://lh6.googleusercontent.com/H7aAEkqf4L0RK6xhTSG4yONGafGKhUvHiSz1SPh9c7kyEPfXmhkwWKtr3twcZt6SMo\_KBWj4JNStxURftlc3qeQ7VCyZXng3gUu6GHTCKjoIgMSYwy2tAcyRP\_KFUImGVrYN2tYC) + +![](https://lh3.googleusercontent.com/Eyo8SV3nasLqqvXPw0zanZopx7sTV0G7SBcuYt63aI4YmZBXW9DgAHalQWfifNFwhTI5e9Qc3SpsfM3MWFHB6oVLBCeAHlkicRQ9FsBEEKnZ6KJZKuNyF7rmIOKVDch-unS4oAFJ) + +{% hint style="info" %} +**Please see:** [**https://productresources.collibra.com/docs/cloud-edge/latest/Default.htm**](https://productresources.collibra.com/docs/cloud-edge/latest/Default.htm) **for more detailed information on Edge installation and configuration** +{% endhint %} + +**1B. Establish Edge’s Connection To Each Data Source (One-Time For Each Source)** + +Additional Steps in Collibra DG include: + +* Provide Connection Name which exactly matches Connection / System Name in Collibra DQ +* Select Connection type e.g. Username / Password JDBC driver +* Input Username and Password to connect to your data source +* Input fully qualified driver class name +* Upload Driver jar (to reduce potential conflicts, use same driver jar from Collibra DQ) +* Input Connection String Input credentials e.g. username / password or Kerberos config file +* Reminder: All of the above information should be the same as in Collibra DQ + +Additional Steps in Collibra DQ include: + +* Verify Connection ‘Name’ in DGC matches Connection ‘Name’ in Collibra DQ +* Verify ‘Connection string’ in DGC matches ‘Connection URL’ in Collibra DQ +* Verify ‘Driver class name’ in DGC matches ‘Driver Name’ in Collibra DQ +* Verify ‘Driver jar’ in DGC matches Driver used in ‘Driver Location’ in Collibra DQ (may require SSH) + +![Where: Settings > Edge > Select Edge site > JDBC Connections > Select ‘Create connection’](https://lh5.googleusercontent.com/-3FpYTn4vo4kWogSJNgPi4afMwty1a8pk-2\_m-bYYTAz195caF4jRbB0OF2bC0U1t559jNLOvXVAgRLt32EpWL5IEjpB8nqUZ0R1A98ODxKmC9GGCavw0Ad5iXTHss0nhCtcsK1W) + +{% hint style="warning" %} +**Important: Connection / System name (in this example, ‘postgres-gcp’) must exactly match the Connection / System Name in Collibra DQ** +{% endhint %} + +**1C. Establish Catalog JDBC Ingestion Capability On Edge (One-Time For Each Data Source)** + +![Where: Settings -\> Edge -\> Capabilities -\> Input Name -\> 'Catalog JDBC Ingestion' ](https://lh3.googleusercontent.com/7To6AMTiyioNVZeZwK9pzi14Y7D1vCbAyRV4vj7iteI0wz30cGJI4jNaXO9gtLDSEwhltZnHwr48-NWSbFYtU9LJot7UBovm6-yyEoURnol-ksZ0F-Q81tRVOwYYKvnzesWKB19s) + +**1D: Configure Destinations For DQ Assets (Rules, Metrics, Dimensions) Within DQ Connector (One-Time)** + +Option A: Create New Destinations + +* Create New Rulebook Domain (suggested domain type) for DQ Rules and DQ Metrics + * Global Create -> Search for and select 'Rulebook' under 'Governance Asset Domain' -> Select desired 'Community' e.g. 'Data Governance Council' -> Input name of Rulebook domain e.g. 'CDQ Rules', 'CDQ Metrics' + +{% hint style="info" %} +Record your domain resource ID e.g. 2xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (can be found in your URL) for Step 1G +{% endhint %} + +* Create New Business Asset Domain (suggested domain type) for DQ Dimensions + * Global Create -> Search for and select 'Business Asset Domain' -> Select desired 'Community' e.g. 'Data Governance Council' -> Input name of domain e.g. 'CDQ Dimensions' + * Record your domain resource ID e.g. 2xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (can be found in your URL) + +Option B: Use Existing Domains from existing Rulebook and Asset domains + +{% hint style="info" %} +Record your domain resource ID e.g. 2xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (can be found in your URL) for Step 1G +{% endhint %} + +![](https://lh3.googleusercontent.com/LOR\_nFMAjSrFUPNoOLqjEvDAxwBoeHsjYU\_c\_QsgH3QgRH-y8\_Uvtn2kfMesE68VcYDrpp1raPKOumsOAyoVek32I2x-e9OskR-JUDV8GxdmtFNOMK6iKTvcDejs56PXpmvh6pa1) + +{% hint style="success" %} +You have now established destinations for where Collibra should ingest your User-Defined Rules, Metrics, and Dimensions +{% endhint %} + +**1E. Assign Permissions for New Domains of DQ Assets (Rules, Metrics, Dimensions) (One-Time)** + +Please assign your Edge user as a 'Technical Steward' in each of the domains specified in 1D such that Edge can write create / update assets into each respective domain + +![](https://lh3.googleusercontent.com/U5NnQT6ncu6XbL7YKWe8A0caumHxodHBCgU\_vJeJjWCyxYSzvGsRWWzH2HjjeVKYih4hvGXvDM7J1\_DB72EgdxoVLrRTNEQsg4enZotY7fEgjfxfI-cbz4vmFHxxzxEBQfUq\_ZpF) + +{% hint style="success" %} +This step provides Edge with the proper permissions to create and update assets into the domains from the previous step +{% endhint %} + +**1F. Allow DQ Assets To Attach To Tables and Column Assets (One-Time)** + +Now we need to add a few relations and update global assignment characteristics + +* **Table**: Settings -> Operating Model -> Relations -> Search in any column for 'Table' -> Global Assignment -> Characteristics -> Edit (larger of the two buttons) on right -> Add characteristic -> Search for and select 'governed by Governance Asset' -> Save + +![](https://lh5.googleusercontent.com/RYKx\_CdwVsVauaJPmjP7yDlZzzTVCeLzcTQHix2wSOmgo8taapeg8lU87L5qatGIrbPKATLMEvN7skj7JHtqGJjMXGuJcVpWD5BToX1W92Q2edGs3ODi3CZ1C4WQX9MMW9bdmKFw) + +* **Column**: Settings -> Operating Model -> Relations -> Search in any column for 'Column' -> Global Assignment -> Characteristics -> Edit (larger of the two buttons) on right -> Add characteristic -> Search for and select 'is governed by Data Quality Rule' -> Save + +![](https://lh5.googleusercontent.com/JhUNVCCoHxkamUgeLsQ98tWPPvHeUNiGB6EjH\_wEugZ5OaenbQbTEL6bVv6W2EiyqtnivaDjCsoHagW0Q6U6s5GDpC37\_vnVND\_qhPGaZDw9GoUry6vrBqiSUwAABfi1npUO18\_E) + +**1G. Establish DQ Connector (One-Time)** + +DQ Connector is an Edge capability that will facilitate communication with your Collibra DQ instance + +* Settings -> Edge -> Capabilities -> Add Capability -> Select 'DQ Connector' -> Input your Collibra DQ URL e.g. 'customerdq.collibra.com:port' input username and password + +{% hint style="info" %} +Remember from previous step 1D, you will need to provide your resource / UUIDs for your specified domains for DQ Rules, Metrics, and Dimensions +{% endhint %} + +![](https://lh4.googleusercontent.com/nRWd59wkPl\_yXKwCsgfvBuFMdiAwlW6nBoN1eV7c2YHN-Y2cHbC82TwGRiub297mQ0uBphUL4vewsBzFOKhesF5gaY6W3Beft2VC4ILmrJZuW8oiqEa45JrvHPFI1QiFtlC4kgs\_) + +**Specify DQ Asset Destinations Within DQ Connector** + +![Input your UUIDs from Step 1D for Rules, Metrics, Dimensions](https://lh6.googleusercontent.com/x-JOYKbeDBzsSnKI5czFiBkUuitcxQeLte9MAsTSJL3sfyw8\_3AwUog9HRyWdjUV9GpVdaUiM199dTf1NfNMGAfiANWmiW93VYAgZs\_PNnogG1KnKa1JRLxJSkEjLrp6J57iQVn-=s0) + +{% hint style="success" %} +Excellent! We've now completed the initial one-time configuration! +{% endhint %} + +## Step 2: Register Edge Connections to Collibra Catalog + +**2A. Create System Asset Within Collibra Catalog To Connect To Edge** + +![Global Create > 'System' > Select Domain > Enter Name e.g. 'postgres-gcp'](https://lh4.googleusercontent.com/e19f1vFffMGQDzgHD1m83C66pG9MwOeJwrd8-Jl42oC9ArjXaCGrwfu\_baSzdP4u1xelfB0YWHYA90tsT9g3NFHIE2ULhIdWnkZRUYi8f1sq8EIltYnm\_BhC-yVDSknI\_9ndGpB0) + +{% hint style="warning" %} +**Important: Connection / System name (in this example, ‘postgres-gcp’) must exactly match the Connection / System Name in Collibra DQ** +{% endhint %} + +**2B. Register Edge Data Source to Collibra Catalog** + +![Catalog > Global 'Create' > Register Data Source with Edge](https://lh6.googleusercontent.com/NsIO-7QVn8gMLJi0YJmhC-gs-r26nPQWwQUY8-S2oQa-pWQAjMeJvo2ZvX5FYG3KqfrbuVE5U5aEeCj25kX19TuDL9MR4ves52EcMyadYgfbWIrC86rHinl7a\_ZUnv2gW9IPRlIZ) + +## **Step 3: Start Ingesting Collibra Data Quality Into Catalog** + +{% hint style="info" %} +**Prerequisite: Catalog will have ingested schemas on Edge** +{% endhint %} + +![Catalog > Data Sources > Select Database e.g. 'postgres' > Configuration](https://lh6.googleusercontent.com/HgMjwe6cR3ne\_GpzqhHQNdB5tMWhIsfg-mU5iLUq7oBZnuomANBVhPGdMSH8kCHBwonZQVp2EhFMQ6H4eH\_P5t7lIGJrboU2y71Hy0HVvenK6uu8PeaxRCSQbEX1LbeKdSlBcdd7) + +{% hint style="info" %} +**Prerequisite: Ensure targeted schemas have User-defined Rules, Metrics, and/or Dimensions within Collibra DQ that have been Executed** +{% endhint %} + +**3A. Synchronize Data Quality for Selected Schemas** + +![Catalog > Data Sources > Select Database > Configuration > Quality Extraction](https://lh4.googleusercontent.com/Xt8y\_PfQ3UZAWBdW6PTWTSX0ZU2830Z-MJykaugTuaWIFIyJR3Hdy0WmijTlFn47yhozmxVe-idXGk7u8wVlfCbk7qIAJMItx44pYVvDIDgjeL62DZ3i38ZrnBjTfwKhB9qa8Irs) + +**3B. Verify Data Quality Results in Collibra Catalog** + +![](https://lh4.googleusercontent.com/syFKZtlFsc0QAv2OwFx10oVNSfgUaA6fe004elBAWo8DXKXDKUdCpsuOyVK5zVNmhKwnYLLQi\_XdKV7B4BcTKLqtov4QCK2b\_MoSHDneKm0abhXv0BE33pQjtOfWb2IE4nJIpWNd) + +{% hint style="success" %} +**Success! Example Output** +{% endhint %} + +**Appendix: Synchronization For Single Table in Data Quality and Data Catalog** + +![View in Collibra Data Quality](<../.gitbook/assets/image (61).png>) + +![View in Collibra Catalog](<../.gitbook/assets/image (62).png>) + +## FAQ + +**Q: Known Limitations** + +* Only 1 source tenant from Collibra DQ can be specified +* On-demand ingestion (vs. scheduled) +* Can only specify 1 domain destination for each of Rules, Metrics, and Dimensions +* Only JDBC sources supported (no file sources) + +**Q: DQ Dashboard In DGC: I can verify the DQ Connector is synchronizing Data Quality Rules and Data Quality Metrics, but why don't Data Quality Dashboard Charts display?** + +A: Ensure correct **Aggregation Paths** and **Global Assignments** (or create, if none exist) for **Table** and **Column** below + +![Aggregation Path For Table (Data Quality Rules)](https://lh5.googleusercontent.com/74HV9oMYQkhBw-jrof1ubunkWPo8OZmgLrxHCM\_J0fFmwS0JR5HoZgCN6-TeGCxyArM65jjjPJSMuUXkaog0qFBnthfOlIDZfZCvjQ-bj7dM\_ALfcnFYENw\_8u4UnJwDnDGlort3=s0) + +![Aggregation Path for Column (Data Quality Rules)](https://lh6.googleusercontent.com/Bo0xlZmY9-EbKc2nkbI8fOFX3IOdI2\_HFkFDt-mw99H10ovIU-nD6m1Jd1pnnfvcHceynzV07NzoarH\_AVbzFg90uyITf5PRM9bTWNHoPXGdVEgmJzR\_MkQrjLVgRhcnTPh3pIj7=s0) + +![Global Assignments For Data Quality Rules](https://lh5.googleusercontent.com/9VHho9WDZhgDLAGYzZ-llIUWFJrLF6mis\_BpDG5HI-I45mVgi5yOF7p74sONmnLZ9e4pY4GaaCyEIl4yCTVInzbBBjgpmPOUV0NQykcdkUmmx6s-6\_DvoTpqs82jMO6AVBsM8\_E1=s0) + +**Q: DQ Dashboard In DGC: Why won't my DQ Dimension charts display in my Dashboard?** + +A: Please 1) add a new custom **Relation** 'Data Quality Metric classified by Data Quality Dimension', 2) **Global Assignment** for 'Data Quality Metric', 3) UUID of the new **Relation** into the **DQ Connector** setup in **Step 1G**, 4) + +![New Relation Type In Operating Model](https://lh5.googleusercontent.com/WqDQjGVtgHTGzDNwrb-yPZlhcAC11vAU2KxHox3QAZ4E1c-ThfbakEO9fjl2ZyqscPLB5a3FjBZvYQxxtYd89uv5YGEgCDTvIBUty9JoMsgTLl-I1dEtqo4vzIFKSas6rtXYpb5T=s0) + +![New Global Assignment Characteristic](https://lh6.googleusercontent.com/9fxsT4RRkyFDh93CyK6ceRpLYTxboAk3XwpxRWtmp5om9ViZutcuvcOzaHFVh-R02n0mSJhhcYzhvptAOeFT3lWK0HKtI\_YLEaH1SUFgXN-5JQz532pdra-fsP2pD4w3XnDvNiEl=s0) + +![Add 'classified by Data Quality Dimension'](https://lh4.googleusercontent.com/Suo299gJGe1Y2tsK6rE4m6kca0XO1Dp7WoHvKfrswRUxxobQBI\_AD0KfZswl5mcEtUVwm\_W70Ws0OJp2-as62s9NfvuqUzBwqsntCATSsCbREHefpFTJ0TsSOlRsnHME210goEIJ=s0) + +![Copy Resource ID of New Relation Into DQ Connector Setup](https://lh4.googleusercontent.com/GNknIUuSsGe\_CiIyA\_eAymSeCvtQvQ7yHoqviqLcFWhF3MNQp5ynx\_r1a3eDg4\_Yw46gyegdJSlymUXxTZ91HOg5y\_xIWbVGSjeXRreA3rXyHofkwLaJzDBZ8ZpgfUcFx-pAqtUN=s0) + +**Q: I've connected and configured data sources correctly, why aren't DQ Rules and DQ Metrics being synchronized?** + +A: Please ensure Connection / System Names between Collibra Data Quality, Collibra, and Edge exactly match + +A: Please ensure Edge user has admin permissions to write the assets into Catalog + +A: Please ensure correct URL specified within the DQ Connector capability e.g. http://cdq.customer.com:9000/ + +**Q: Is DQ Connector unidirectional?** + +A: Yes, from Collibra DQ to Collibra Catalog in Data Intelligence Cloud + +**Q: How many DQ Connectors can I run simultaneously?** + +A: Currently, one. + +**Q: Does the DQ Connector work with On-Prem Collibra DGC?** + +A: No, any work with on-prem Collibra DGC would be custom API development via Collibra Professional Services or a partner SI. + +**Q: If I delete a rule from Collibra DQ that I have already synchronized into Collibra Catalog, will it be deleted from Catalog in the next synchronization?** + +A: No, the DQ Connector only upserts into Catalog. If a rule is deleted from Collibra DQ, it will not be automatically deleted in Catalog. + +**Q: Why are my scores different in Collibra DQ and Collibra Catalog?** + +A: Currently, the DQ DQ Connector pulls in the most recent user-defined rules from Collibra DQ. Other components that affect score such as Behaviors, Outliers, Patterns, Dupes, Source are not yet included. + +**Q: Getting errors when trying to delete both domain that Edge created for DB and the Connection?** + +A: Please delete Edge created domain via API. + +**Q: I've hit the synchronize button, how can I tell if my job is complete?** + +A: Check the Activities circle (button on top right of menu) for the status of your DQ Synchronization. diff --git a/integration/dq-workflows-in-collibra-dic.md b/integration/dq-workflows-in-collibra-dic.md new file mode 100644 index 00000000..2974eb02 --- /dev/null +++ b/integration/dq-workflows-in-collibra-dic.md @@ -0,0 +1,118 @@ +# DQ Workflows + +## Benefits + +The DQ Workflows package listed on Collibra Marketplace allows you to 1) create and manage Data Quality Issues, 2) receive Notifications on Rule Metrics, and 3) request Rule Creation and Modification within **Collibra Data Intelligence Cloud**. Data stewards will be able to organize and prioritize all requests within DIC before they take any action within **Collibra Data Quality**. + +Once deployed, the workflows will facilitate quicker data issue remediation by involving business analysts and other personas who can now participate in your data quality workstreams. + +_Please note: DQ Workflows are listed on Collibra Marketplace and are templates to get customers started. Collibra-provided Marketplace listings are not subject to the same SLA obligations (_[_https://marketplace.collibra.com/marketplace-terms/_](https://marketplace.collibra.com/marketplace-terms/)_) In addition, they can only be leveraged within Collibra Data Intelligence Cloud. In the future, we will work towards releasing bi-directional workflows._ + +## Step 0: Prerequisites + +| | | +| -------------------------------- | ----------------------------------------------- | +| **Resource** | Notes | +| Collibra Edge Site | DQ Connector is a capability of Edge | +| Collibra Data Intelligence Cloud | 2021.07 Release (or newer) | +| Collibra Data Quality | 2.15 (or newer) | +| Collibra DQ Connector | Synchronized Rules from Data Quality to Catalog | + +{% hint style="success" %} +Let's proceed after gathering all prerequisites! +{% endhint %} + +## Step 1: Download, Deploy and Start DQ Workflows + +**1A. Download Package from Collibra Marketplace and Unzip Files** + +\*\*1B. Deploy Workflows \*\* + +![](<../.gitbook/assets/image (68).png>) + +**1C. Adjust Workflow Settings (One-Time Setup)** + +| Workflow Configuration Setting | DQ Rule / DQ Sync Request | DQ Rule Modification | DQ Data Remediation | DQ Issue Resolution | Manage DQ Subscriptions | Notify of DQ Metrics | +| ------------------------------------------------------------------------ | ------------------------- | -------------------------------- | ------------------- | ------------------- | ----------------------- | -------------------- | +| Applies To | Asset | Asset | Asset | Asset | Global | Global | +| Applies To Asset Type | Column, Table | Column, Table, Data Quality Rule | Column, Table | Issue | | | +| Other: Any Signed In User Can Start Workflow | Y | Y | | | Y | Y | +| Other: Perform Candidate User Check on Workflow Start | Y | Y | Y | | Y | Y | +| Other: This Workflow Can Only Run Once At Same Time on Specific Resource | | | | | Y | Y | +| Other: Show In Global Create | | | | | Y | Y | +| Roles: Start Workflow | | | | | Sysadmin | Sysadmin | +| Roles: Stop Workflow | | | | | Sysadmin | Sysadmin | +| Roles: Reassign Tasks | | | | | Sysadmin | Sysadmin | + +![Example: DQ Rule Request](<../.gitbook/assets/image (69).png>) + +![Example: Manage DQ Subscriptions](<../.gitbook/assets/image (70).png>) + +## **Step 2: Create Data Quality Requests / Issues** + +### **2A. Create Data Quality Issues** + +| Workflow | Main Requestor Persona | Description | Steward Taking Action | +| -------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | +| DQ Data Remediation |

Data Steward

Business Analyst

| Tracking / management for confirmed data issues which may require underlying data remediation | Data Lake Admin, ETL Engineer | +| DQ Rule Request | Business Analyst | Proposing data quality rules in plain language e.g. "flag any German phone numbers in this dataset" or "identify customers with churn risk based on engagement time with our platform" | Data Steward | +| DQ Rule Modification | Business Analyst | Proposing adjustments to existing rules e.g. values, dimensions, passing thresholds | Data Steward | +| DQ Sync Request | Business Analyst | Request for synchronization of the DQ Connector to synchronize and/or onboarding a new dataset with pre-populated rules | Data Steward | + +![Example: Data Quality Rule Request](<../.gitbook/assets/image (140).png>) + +![Completed Rule Request Example](<../.gitbook/assets/image (104).png>) + +![Example: Data Quality Rule Modification Request](<../.gitbook/assets/image (131).png>) + +![Completed Data Quality Rule Modification Example](<../.gitbook/assets/image (130).png>) + +![Example: DQ Synchronization Request with Rules Request optional checkbox](<../.gitbook/assets/image (150).png>) + +## Step 3: Manage Data Quality Issues + +### 3A. Setup Data Helpdesk Filter + +**Data Helpdesk** + +* Select Issues +* Navigate to 'Filters' +* Properties > Attributes > Relations > Issue \*\*categorized by \*\*Issue Category > Input 'Data Quality Issue' > Apply +* Save button > Save View as > '**Data Quality Issues**' +* Optional settings for View: Can pin, promote, make public, make default + +### 3B. Manage Issues From Data Helpdesk View + +![](<../.gitbook/assets/image (71).png>) + +### 3C. Alternate: Manage Issues From Tasks + +![](<../.gitbook/assets/image (72).png>) + +## Step 4: Receive Notifications Of DQ Issues And Metrics + +### 4A. Set Up DQ Metric Subscription + +Who? Anyone can set up a DQ subscription, for yourself or for your teammates + +![](<../.gitbook/assets/image (149).png>) + +Alerts will be sent based on reviewing rules and metrics associated with **Tables** or **Columns** that violate the specified **Threshold** + +![](<../.gitbook/assets/image (112).png>) + +Assuming an e-mail is associated with the Subscriber within Collibra, the Subscriber will receive e-mail notifications by default at **12pm local server time**. This, along with other settings within the provided workflow, can be **adjusted** in **Eclipse**, Collibra's recommended workflow editor. + +### 4B. Review DQ Metric Alerts + +Ensure that the DQ alerts set for you are providing helpful details + +![](<../.gitbook/assets/image (136).png>) + +### 4C. Update Subscription Settings + +![](<../.gitbook/assets/image (124).png>) + +For every subscription set up for a Subscriber, the Manage DQ Subscriptions modal will cycle through for your review. You can update **Threshold**, add or delete **Notification Days**, add or delete **Tables** or **Columns**, rename the **Subscription title**, **Save the new settings**, or simply **Unsubscribe**. + +![](<../.gitbook/assets/image (118).png>) diff --git a/labeling-training/item-labeling.md b/labeling-training/item-labeling.md new file mode 100644 index 00000000..5f01b5cf --- /dev/null +++ b/labeling-training/item-labeling.md @@ -0,0 +1,33 @@ +--- +description: Quickly click on items to trigger re-training +--- + +# Item Labeling + +{% hint style="warning" %} +Note: There are limitations with item labeling. Not all line items are eligible for all labeling options (Validate, Invalidate, Resolve). Please see the breakdown below on which options are available per DQ detection tab. Some of this is by design. Additionally, we are normalizing the actions and updating documentation for clarity. +{% endhint %} + +![](../.gitbook/assets/item\_label.gif) + +## Re-train the Owl DQ Model + +By clicking the down arrow, you are telling Owl that this finding is not an actual DQ item. Another way to say it is "I don't want to be alerted to this item in the future". Owl allows a user to annotate the reasoning while keeping an audit log of the event. You can bring the item back into play by navigating to the "Labels" panel and deleting the down trained item. Owl will prompt a user to re-train the model after clicking a down arrow. + +``` +Behaviors - Validate, Resolve (You cannot invalidate something that is generated) +Rules - NONE (Managed through Rules Interface) +Outliers - Validate, Invalidate, Resolve +Pattern - Validate, Invalidate, Resolve +Source - Validate, Invalidate, Resolve +Record - Validate, Resolve +Schema - Validate, Resolve +Dupes - Invalidate Only +Shapes - Validate, Invalidate, Resolve +``` + +Users can assign items or label for future runs. + +![Each item will have an action to assign or dismiss the item.](<../.gitbook/assets/image (141).png>) + +![](<../.gitbook/assets/image (122).png>) diff --git a/labeling-training/peak-vs-off-peak.md b/labeling-training/peak-vs-off-peak.md new file mode 100644 index 00000000..51ef1e3e --- /dev/null +++ b/labeling-training/peak-vs-off-peak.md @@ -0,0 +1,15 @@ +# Peak vs Off Peak + +![](../.gitbook/assets/off\_peak.gif) + +## But my Weekend runs are not the Same + +A common scenario that can fool behavioral analytics and ML is when you have 2 or 3 different but normal patterns. Owl has a rich labeling system that allows a user to fork the training model to learn these cycles individually as to not confuse the model. + +![](../.gitbook/assets/owl-rich-labeling.png) + +## Click the Orange Button + +By clicking the orange button you can label the day of the week as peak vs off peak. You can also chose your time zone - this will help determine the day of the week accurately. A user only needs to click the peak scheduler once and the model will learn and forecast this understanding for every run in the future. This feature commonly prompts for a re-train. + +![](../.gitbook/assets/owl-off-peak.png) diff --git a/projects/best-practices/README.md b/projects/best-practices/README.md new file mode 100644 index 00000000..15655d61 --- /dev/null +++ b/projects/best-practices/README.md @@ -0,0 +1,73 @@ +# Best Practices + +### **Understanding owl activities and what the key/date columns mean for each** + +* Starting with profile and expanding to rules and then other advanced capabilities. + * [https://dq-docs.collibra.com/dq-visuals/profile](https://dq-docs.collibra.com/dq-visuals/profile) +* Training with Owl-team Zoom/On-site support +* Running with sample data +* Introducing anomalies on sample data and running an owlcheck to see the anomalies. + +### **Using the tool with practical scenarios** + +* Having Well Defined Use Cases + * Determine a single table (dataset) that you would like to scan + * Have an expectation of what you would expect Owl to find in this dataset + * Understand which activities would capture the expected findings +* Target internal datasets with known data issues +* Historical Comparisons: + * If pre-cleaned data is available with data findings that have been cleaned via legacy methods such as internal rules, run these datasets and compare the results from Owl to Internal findings. +* Work with data owners to understand findings or review expected findings + +### Explorer + +* The date selected with the calendar widget in the Scope (home) tab should align with the calendar widget assigned on the final (Save/Run) tab. +* If you elect to Unlock the cmd line and override the final parameters, do not re-lock or the changes will be overwritten. In general, only advanced users should override the guided settings. +* Pushdown and parallel JDBC cannot be used together. If you are using pushdown, do not select the parallel JDBC option + +### Files + +* File paths should not contain spaces or special characters. +* Backrun (replay) and advanced features are best suited for JDBC connections. Some features are unavailable if file and storage naming conventions do not consistently contain a date signature. + +### Connection Pool + +If you see this message, update the agent configs in owl-env.sh or agent confg map for k8 deployments. + +``` +Failed to obtain JDBC Connection; nested exception is org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-29-thread-2] Timeout: Pool empty. Unable to fetch a connection in 0 seconds, none available[size:2; busy:1; idle:0; lastwait:200]. +``` + +Adjust these configs to modify the connection pool available. + +``` +export SPRING_DATASOURCE_POOL_MAX_WAIT=500 +export SPRING_DATASOURCE_POOL_MAX_SIZE=10 +export SPRING_DATASOURCE_POOL_INITIAL_SIZE=5 +``` + +### Freeform Agent Configs + +When configuring the DQ Agent and using the Free Form Parameters at the bottom of the dialogue, you need to comma separate multiple -conf key/value pairs. I am going to write this as a forum post but use this format: "-conf some.key=x, some.other.key=y" + +### K8 Secrets + +The following Env Vars are now managed as a Secret instead of as a Configmap + +LICENSE\_KEY LIVY\_SSL\_KEY\_PASS SERVER\_SSL\_KEY\_PASS SPRING\_AGENT\_DATASOURCE\_PASSWORD SPRING\_AGENT\_DATASOURCE\_USERNAME SPRING\_DATASOURCE\_PASSWORD SPRING\_DATASOURCE\_USERNAME + +### DQ Job Stages + +DQ job failure is one of the most frequently asked. This outlines the DQ Job Lifecycle and where to find logs for each phase. Every DQ Job goes through a three stage Lifecycle: + +#### Stage 1 + +Agent picks up job from the Metastore and translates it into a valid Spark Submit request. This includes credential acquisition and injection for Cloud and Kerberos. If a job never makes it out of STAGING, the first thing to do is to check the Agent logs (\/log/agent.log or on K8s kubectl logs -n . + +#### Stage 2 + +Agent hands off the DQ check to Spark via Spark Submit, maintaining a handle on the Spark Submit request. At this point the Job is in Spark’s custody but not yet running (Spark Submit creates its own JVM to manage the submission of the Spark Job to the cluster/runtime). If the job fails with a message saying something like “Failed with reason NULL” on the Jobs page, check the Stage 2 logs (there will be a separate log for each Job). These can be found either on the Agent itself (\/log/.log) or whenever possible on the Jobs page Action Dropdown on the job entry. Stage 3: Spark Submit instantiates the Job in the target Spark Runtime (Hadoop/K8s/Spark-Master). At this point, the DQ core code is active and DQ is back in control of the job. Typically, if a job makes it to this stage, it will no longer be in STAGING status and you should see an error message on the Jobs Page. Typically, the full Stage 3 log is required to trouble shoot a problem that happens in Core. + +#### Stage 3 + +logs can be obtained from the Actions drop down for the job entry. If log extraction failed, job logs will need to be gathered from the Spark Runtime directly (Hadoop Resource Manager, K8s API via Kubectl or vendor provided UI, Spark Master UI or directly from the Spark Master Host). diff --git a/projects/best-practices/prescriptive-personas.md b/projects/best-practices/prescriptive-personas.md new file mode 100644 index 00000000..62b7d867 --- /dev/null +++ b/projects/best-practices/prescriptive-personas.md @@ -0,0 +1,7 @@ +# Prescriptive Personas + +Prescriptive Personas + +Owl has 4 prescriptive personas to manage user permissions, Analyst, IT Admin, Observer, and Steward. Simply click on the user icon located on the bottom left of the blue Owl Menu bar and select user profile. The persona type can be assigned under the access tab in profile management. + +![](<../../.gitbook/assets/Profile Management.jpg>) diff --git a/projects/data-projects/README.md b/projects/data-projects/README.md new file mode 100644 index 00000000..1ca77b7e --- /dev/null +++ b/projects/data-projects/README.md @@ -0,0 +1,2 @@ +# Data Projects + diff --git a/projects/data-projects/assists-data-aggregation.md b/projects/data-projects/assists-data-aggregation.md new file mode 100644 index 00000000..7cd28224 --- /dev/null +++ b/projects/data-projects/assists-data-aggregation.md @@ -0,0 +1,7 @@ +# Assists Data Aggregation + +* **Greatly reduce Support Costs.** Apply AI-generated rules to maintain consistency across all accounts, rather than apply hand-made rules per every account. You are not managing 100+ variations of rules, but a consistent set that is learned from all. +* **Capitalize on Real-time.** Batch collection with excel compare will never support real-time. Instead, rules are generated by the data itself, and anomalies are triggered on the fly at the source system. +* **Rapid Onboarding.** Universal and already tested Scans are applied quickly. +* **Improve Customer Satisfaction.** Monitor the real-time speed of the DQ pushdown fix, not just the overall DQ score over time. +* **Improved SLAs.** When DQ is fixed immediately, all SLAs can be improved not just DQ SLAs. \ No newline at end of file diff --git a/projects/data-projects/builds-a-better-dq-dashboard.md b/projects/data-projects/builds-a-better-dq-dashboard.md new file mode 100644 index 00000000..8011b2a1 --- /dev/null +++ b/projects/data-projects/builds-a-better-dq-dashboard.md @@ -0,0 +1,6 @@ +# Builds a Better DQ Dashboard + +* **DQ Dashboards.** Many DQ problems result from an improper or a too slow observation of business rules related to the data. What is not caught by handmade visual inspection or a potentially outdated man-made rule can only be flagged by AI Machine Learning. Conversely, what does get flagged should also be easily triaged and then immediately fixed with the aid of AI. The most important metric for a DQ Dashboard is the time to fix, not simply the overall DQ score. + +{% embed url="https://youtu.be/Lu2I7n_nRlc" %} + diff --git a/projects/data-projects/ccpa-and-gdpr.md b/projects/data-projects/ccpa-and-gdpr.md new file mode 100644 index 00000000..9216a78e --- /dev/null +++ b/projects/data-projects/ccpa-and-gdpr.md @@ -0,0 +1,7 @@ +# Ensures CCPA & GDPR + +Aggregation from 100s of locations. The dashboard for what is within spec based on AI observation – not handmade rules. + +Then push-down fix. The rules are created and then immediately applied via self-service. The problem is immediately identified and a fix (recommendation engine) is applied. + +The Value of this for both companies. The DQ problem never corrupts the whole. The longer the bad quality exists the bigger problem it can create. \ No newline at end of file diff --git a/projects/data-projects/creating-a-data-quality-pipeline.md b/projects/data-projects/creating-a-data-quality-pipeline.md new file mode 100644 index 00000000..24679a02 --- /dev/null +++ b/projects/data-projects/creating-a-data-quality-pipeline.md @@ -0,0 +1,51 @@ +# Creating a Data Quality Pipeline + +Organizations who leverage data as an asset in order to make decisions into their future have to entrust the data in which important business decisions are derived from. While almost all businesses leverage their own collected or generated data (or plan to) for internal use, I question how many actually scrutinize their data? Companies who sell a product have to make sure the product is run through some sort of quality assurance suite of tests/runs before it lands in the hands of a customer. Well much the same, organizations who use data internally as their own asset/product should have the same or more confidence in the quality of their data. Owl-Analytics is a data quality product that observes the data in order to surface behaviors, patterns, outliers, schema changes, record changes, and more. + +Data Scientists are trying to find insights in data to solve complex problems for their business. However, 80% of their time is spent discovering the data in order to cleanse the data to make it ready for the model. Over time models deteriorate as data underneath changes or new trends/ patterns arise. Leveraging Owl-Analytics to validate the quality of the data during the data pipeline and before the data is presented to the Data Scientist will reduce time to value for business insights as Data Scientists will get time back not cleaning / prepping the data and helping the model maintain a longer life. + +Azure Databricks Allows the ability for Scala code to be written in a Jupyter Notebook against an Azure backed Databricks cluster in order to scale the work out to more nodes in order to support the model and the amount of data being crunched by a business’s data scientists. The simplistic nature of Azure and Databricks along with the unification of Spark and Jupyter Notebooks on top of a robust infrastructure from storage to compute allows for Owl-Analytics Data Qualified pipelines to be built and executed seamlessly reducing the time to valuable insights. + +Here is how you can build such great DQ pipelines……… + +### Step 1: Build a Databricks Cluster in Azure. + +Within the Azure portal find Azure Databricks Service and create a cluster, after the cluster is built you should be able to launch the Workspace as shown below. + +![Figure 1: Azure Databricks cluster creationStep](https://firebasestorage.googleapis.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Lb9zlrkw1AEC\_k2s2wG-2632701976%2Fuploads%2FWdz8iiZJLOBRp8nQTdB0%2Ffile.png?alt=media) + +### Step 2: Create a Cluster, add the Owl jar and create a Notebook. + +1.) Inside the Azure Databricks UI first create a cluster providing a cluster name (in this example we will be using DBOWL2 as the cluster name) and selecting the Databricks Runtime Version that Owl current supports (as of this blog post) which is Runtime: 5.2(Scala 2.11, Spark 2.4.0). + +2.) After the Cluster is created make sure to import owl’s jar file onto the cluster so that theNotebook can access the methods exposed in the jar file. + + 3.) Now that the Jar file has been added, create the Notebook and attach the notebook to the cluster. + +![Figure 2: Import owl-core.jar and create new notebook](../../.gitbook/assets/image.png) + +Now the cluster is running with the Owl jar loaded on the cluster. Open the Jupyter notebook attached to the cluster and begin looking at a data set as a Data Engineer would, prepping the data for use by a Data Scientist by leveraging a DQ Pipeline as shown in the below screen shot. + +\ + + +![Figure 3: Notebook on Azure Databricks to scan raw data](<../../.gitbook/assets/image (1).png>) + +This Scala code imports the Owl jar and loops through the dates residing in files on Azure blob storage, pulls them into a Spark Data Frame (DF), and execute an Owl job to scan for the quality issues on the Spark DF. Once the scan is completed the results are stored into the metadata repository under Owl’s web application and visible through your browser as shown in Figure 4 below. + +![Figure 4: Scorecard displayed in the output of Owl scanning the DF](<../../.gitbook/assets/image (2).png>) + +The reason for a score of 49 on the raw data (as shown below in Figure 5) is due to the file having string values sprinkled in the file when something is Not Applicable (N.A.). When reading data in a column of a file that has a mix of numeric and string values the column will automatically conform to a string regardless if the majority class are integers. Also, within the files there is a single record in this file that has meta data information about the file “META\_ZZ” this is also adding empty strings for all other columns. This record will also cause all columns to conform to strings. + +![Figure 5: Data Preview of the raw file content some reasons why Owl scored so low.](<../../.gitbook/assets/image (3).png>) + +Now that we have an understanding of the raw file and how we need to conform it before analysts can start to glean business value from the contents itself. We first have to ETL or cleanse the data that we discovered as being errand by filtering out the erroneous record and flipping all the N.A. values to null as the next step in our ETL and DQ pipeline. + +![ Figure 6: Spark ETL / DQ Pipeline](<../../.gitbook/assets/image (4).png>) + +The Owl block of code is essentially the same, however there is a new owl property added to auto filter values “props.nullValue = ‘N.A’”. This will find every cell that has the value of N.A. and conform it to a “null”. Once the file is read into a Spark DF then we use Spark to “Filter” out the erroneous record on line 36 in the code snippet above. Notice we are also adding an Owl\_Run\_ID date as this dataset did not have a date that conforms easily. After the ETL process cleanses the data, we then have Owl’s Data Quality engine scan the newly processed Spark DF storing the results into a dataset called CleanCSVFiles (as shown in Figure 7 below). + +![Figure 7: Stack the DQ Scores in Owl to show how we did ](<../../.gitbook/assets/image (5).png>) + +Notice the composite scores in the boxes are substantially better for the CleanCSVFiles dataset than what they are for the original RawCSVFiles. In the next article we will look deeper at the intelligence an Owl scan can garner on a data set when run over several days and how Owl can surface different patterns, behaviors, trends and more in the data itself. + diff --git a/projects/data-projects/data-quality-monitoring.md b/projects/data-projects/data-quality-monitoring.md new file mode 100644 index 00000000..12c3f573 --- /dev/null +++ b/projects/data-projects/data-quality-monitoring.md @@ -0,0 +1,26 @@ +--- +description: Project-Based Use Cases +--- + +# Makes your Data Lake better. + +Data Lakes support analytics which will ultimately drive actions which increase revenue, support compliance, prevent churn, etc. However, whether that action is near to real-time or not, none of those can be performed without first performing a DQ check. For example, can you trigger an action before first checking the “GDPR Remove” list? A Data Quality check must always be the first step in any action. OwlDQ with Schema Learned can perform 100+ owl checks. However beyond simply those checks, it is OwlDQ's unique Spark-based architecture listed below that enables innovation. Churn, credit check, AML, infosec checks developed in the Data Lake could be added as part of Owlcheck on the streaming data. + +* **Data and Privacy in Place.** Data never has to move for a DQ OwlCheck. The latency saved from operating in place, the added hybrid flexibility, the privacy maintained serves many new use cases that were not possible before. It also removes any unnecessary consolidation for the sake of simply consolidation. DQ doesn't have to start by first moving it into a Data Lake. + + +* **DQ or Any Rules applied in the Stream.** The DQ rules learned by Owl can be applied back to the source on data in the stream. However, other non-DQ rules learned in the Data Lake can also be added to the OwlDQ check. + + +* **Self-Service and DQ push-down fix.** Owl can enable a self-service push-down fix (recommendation engine) to anything flagged at the source. The best time to fix DQ is when and where the problem started. This enables tighter integration with Data Governance tools since DQ is maintained at the source once, not downstream where corruption beyond just the data can occur. + + +* **Multi-cloud/On-prem/Hybrid.** OwlDQ can scan/alert/report at the source or can operate natively on the target Data Lake such as Databricks Delta in Azure or Snowflake on AWS, or Qubole on GCP. Why compromise DQ just because your data is not in one place? Why settle on a DQ strategy that only works if the data is first migrated or moved? + + +* **DQ Dashboards.** Many DQ problems result from an improper or a too slow observation of business rules related to the data. What is not caught by handmade visual inspection or a potentially outdated man-made rule can only be flagged by AI Machine Learning. Conversely, what does get flagged should also be easily triaged and then immediately fixed with the aid of AI. The most important metric for a DQ Dashboard is the time to fix, not simply the overall DQ score. + + + + + diff --git a/projects/data-projects/migrations.md b/projects/data-projects/migrations.md new file mode 100644 index 00000000..1351c20c --- /dev/null +++ b/projects/data-projects/migrations.md @@ -0,0 +1,6 @@ +# Speeds Migrations/Enables Replications + +* **Speed Migrations/Enable Replications.** Batch collection with subsequent excel compare is very problematic. Instead, rules are generated by the data itself, and anomalies are triggered on the fly at the source system. This type of schema learned approach will speed migration and enable replications with much less overhead. + +{% embed url="https://www.youtube.com/watch?v=-jhtQcZKaIw" %} + diff --git a/projects/use-cases/README.md b/projects/use-cases/README.md new file mode 100644 index 00000000..c8ef0e82 --- /dev/null +++ b/projects/use-cases/README.md @@ -0,0 +1,2 @@ +# Use Cases + diff --git a/projects/use-cases/bank-loans.md b/projects/use-cases/bank-loans.md new file mode 100644 index 00000000..6104ed52 --- /dev/null +++ b/projects/use-cases/bank-loans.md @@ -0,0 +1,96 @@ +# Bank Loans + +It is common for banks to lend money in return for monthly payments with interest. However to do so a bank must make sure that the applications are valid and wellformed to begin the underwriting and approval process. Below we will apply some basic lending concepts to OwlDQ. + +1. Credit Score Validation +2. SSN Validation +3. Loan to Value Validation +4. Interest Rate Validation +5. Duplicate Loan Applications +6. Loan Amount Validation +7. Loan Completeness Validation + +![](../../.gitbook/assets/bank-loan1.jpeg) + +![](<../../.gitbook/assets/Screen Shot 2020-03-30 at 2.35.16 PM.png>) + +## 1. Credit Score + +| Business Check | OwlDQ Feature | Manual vs Auto | +| -------------------------------------------------------------------------- | -------------- | --------------------------------- | +| Is the credit score a whole number | BEHAVIOR | AUTO | +|

Is the credit score within a valid range

(between 300 - 850)

| RULE | credit\_score between 300 and 850 | +| Is the credit score NULL or Missing | BEHAVIOR | AUTO | + +## 2. SSN Validation + +| Business Check | OwlDQ Feature | | +| ------------------------------------ | ------------- | ------------------------------ | +| Is a valid formatted SSN | RULE | AUTO-SSN detection | +| SSN is PII | SENSITIVITY | AUTO-SSN labeled | +| Is the SSN NULL or Missing | BEHAVIOR | AUTO | +| Does the SSN belong to the Applicant | PATTERN | SSN -> first\_name, last\_name | + +## 3. Loan to Value + +| Business Check | OwlDQ Feature | | +| ------------------------------------------------------------------------- | ------------- | ------------------------- | +|

Is Loan amount and

asset value (home or auto) valid numbers

| BEHAVIOR | AUTO | +| 95% loan to value ratio to approve | RULE | loan / asset\_value < .95 | + +## 4. Interest Rate + +| Business Check | OwlDQ Feature | | +| --------------------------------------------------------------------------------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------- | +|

Interest rate between

min and max allowable range

for the loans credit rating.

| RULE COMPLEX |

loan l join rates r on l.credit_rating = r.credit_rating

where l.rate between r.min_rate and r.max_rate

| + +## 5. Duplicate Loan Applications + +Can't give someone the same loan twice! + +| Business Check | OwlDQ Feature | Manual vs Auto | +| ----------------------------------------- | ------------- | ------------------------------- | +| Ensure we don't issue the same loan twice | DUPE | first\_n, last\_n, SSN, Address | + +## 6. Loan Amount + +| Business Check | OwlDQ Feature | Manual vs Auto | +| ---------------------------------------------------------------------------------- | ------------- | -------------------------------------- | +| Loan Amount within lendable range | OUTLIER | AUTO | +|

Loan Amount within lendable range

only lend money between 50K and 3M

| RULE | loan\_amount between 50000 and 3000000 | + +### Resulting OwlCheck + +```bash +-lib "/home/opt/owl/drivers/postgres" \ +-cxn postgres-gcp \ +-q "select * from public.loan_risk_grade where last_pymnt_d = '2019-04-01'" \ +-key member_id -alias loan_risk \ +-ds public.loan \ +-rd "2019-04-01" \ +-dl -loglevel INFO \ +-h 10.142.0.29:5432/owltrunk \ +-numexecutors 10 -executormemory 1g -drivermemory 4g \ +-master yarn -deploymode cluster \ +-sparkprinc user2@CW.COM \ +-sparkkeytab /tmp/user2.keytab -tbin MONTH \ +-dupe -dupeinc purpose -fpgon -fpgkey grade \ +-fpginc grade,sub_grade -fpglb 365 -fpgdc last_pymnt_d \ +-record member_id -dupecutoff 60 -dupepermatchupperlimit 99 +``` + +## Which components did we use? + +We made use of Profiles, Duplicates, Outliers and Rules in this example. The experiments were automatically cataloged and put on a job scheduler. The next time a loan issue arises we will be able to take remediation action using the workflow Q. Over time we can see how the bank loan program is running via the report section. + +![](../../.gitbook/assets/owldq-framework-li.png) + +## Files that can be used to replicate this example + +{% file src="../../.gitbook/assets/interest_rates.csv" %} +Interest Rates CSV +{% endfile %} + +{% file src="../../.gitbook/assets/Owl Dataset (2).csv" %} +Loan Data CSV +{% endfile %} diff --git a/projects/use-cases/bloomberg-data.md b/projects/use-cases/bloomberg-data.md new file mode 100644 index 00000000..3e126ec0 --- /dev/null +++ b/projects/use-cases/bloomberg-data.md @@ -0,0 +1,9 @@ +--- +description: Pattern Mining on Bloomberg data +--- + +# Bloomberg Data + +Owl finds over 50 data quality issues per day in common market data through pattern mining. Sequential pattern mining is a tool that finds statistically relevant patterns between data examples where the values are delivered in a sequence. + +![Identify improbable cross-column anomalies. In this case, the pricing source deviates from historical normalcy.](<../../.gitbook/assets/image (94).png>) diff --git a/projects/use-cases/copying-or-moving-data.md b/projects/use-cases/copying-or-moving-data.md new file mode 100644 index 00000000..b781845a --- /dev/null +++ b/projects/use-cases/copying-or-moving-data.md @@ -0,0 +1,73 @@ +--- +description: Validate Data Integrity Between Distinct Storage Systems +--- + +# Validating Data Movement + +## Record-for-Record Reconciliation + +When you’re copying or moving data between distinct storage systems such as multiple HDFS clusters or between non-HDFS storage and cloud storage, it’s a good idea to perform some type of validation to guarantee data integrity. This validation is essential to be sure data wasn’t altered during transfer. + +Detect potential data corruption caused, for example, by older versions of drivers, parsing errors, connection limits, noisy network links, memory errors on server computers and routers along the path, or software bugs (such as in a library that customers use). + +### Common Data Copying/Movement Scenarios + +* Landing, Loading, Persisting third-party files + * Landing daily files. + * Loading daily files into staging location. + * Finally, persisting data in lake or warehouse. +* Cloud Migrations + * Between existing database storage to optimized cloud storage formats. + * Between local file systems and cloud relational database +* Data Lake or Data Warehouse + * Migrating data from single storage system to distributed storage + * Consolidating storage systems to a single lake or warehouse +* Same Storage, Different Environments + * Copying same data between Dev, QA, and Prod environments. + +{% hint style="warning" %} +_How do you easily validate the same data exists in distinct locations?_ +{% endhint %} + +### Shortcomings of Existing Validation Checks + +* Low-level integrity checks like row counts and column counts may not be sufficient. +* No easy way to reconcile between across non-HDFS files and database. +* Chunk verification requires storage size, format, and metadata to be exactly equal. +* Different data types in two distinct databases (Oracle and Teradata) will not reconcile. +* Two different copies of the same files in HDFS, but with different per-file block sizes configured. +* Two different instances of HDFS with different block or chunk sizes configured. +* Copying across non-HDFS [Hadoop-compatible file systems](https://wiki.apache.org/hadoop/HCFS) +* https://wiki.apache.org/hadoop/HCFS +* (HCFS) such as Cloud Storage. + +Explicit end-to-end data integrity validation adds protection for cases that may go undetected by typical in-transit mechanisms. + +### Enter, OwlDQ Integrity Validation! + +To ensure and protect against target systems getting out of sync or not matching the originating source, turn on `-vs` to validate that the source matches the target. [Read More](https://docs.owl-analytics.com/dq-visuals/validate-source) + +{% embed url="https://docs.owl-analytics.com/dq-visuals/validate-source" %} + +Complete row, column, conformity, and value checks between any two distinct storage systems. Can be run against high-dimension or low-dimension datasets. Works between Files and/or Database storage, On-premise, or across Cloud environments. + +### **Get Started Today** + +We don’t want you to get stuck writing a bunch of reconciliation checks we’ve already written! Focus on other stuff that actually moves your project forward. + +For more information, please contact **info@owl-analytics.com** or **schedule a demo at** [**www.owldq.com**](http://www.owldq.com) **** + + + + + + + + + + + + + + + diff --git a/projects/use-cases/cyber-anomalies-in-real-time.md b/projects/use-cases/cyber-anomalies-in-real-time.md new file mode 100644 index 00000000..09af902b --- /dev/null +++ b/projects/use-cases/cyber-anomalies-in-real-time.md @@ -0,0 +1,80 @@ +# Cyber Anomalies in Real-Time + +With increasing number of cyber threats most of the cyber security team doesn’t have the capacity to manually detect, monitor, and defend against all. Effective cyber threat management requires leveraging automation to inform decisions. + +OwlDQ framework, provides organizations the ability to load and process diverse security data feeds at scale in order to detect network data anomalies. The OwlDQ alerts can enable network admins to respond to these events in timely manner. + +Here we walk through a scenario to detect anomalies with network traffic dataset. + +1. IP address Validation +2. Detect the unusual network traffic patterns based on locations. +3. Identify the suspicious packets based on size. +4. Detect the malicious activity based on source and destination IP addresses. + +![](../../.gitbook/assets/cyber-anomaly-detection-4.png) + +## Infosec dataset Preview + +Dataset contains Timestamp, Source Workgroup, Source IP, Source Port, Destination Workgroup, Destination IP, Destination Port, Application, Protocol and Packet size information. + +![](../../.gitbook/assets/sample\_Infosec\_Data.png) + +## IP Address format Validation + +| Business Check | OwlDQ Feature | | +| --------------------------------- | ------------- | ----------------- | +| Is a valid formatted IP | RULE | AUTO-IP detection | +| Is the IP address NULL or Missing | BEHAVIOR | AUTO | + +## Source and Destination Workgroups + +| Business Check | OwlDQ Feature | | +| -------------------------------------------------- | ------------- | ------------------------------------------- | +| Does it a usual network traffic based on locations | PATTERN | Source\_Workgroup -> Destination\_Workgroup | + +## Source and Destination IP Address validation + +| Business Check | OwlDQ Feature | | +| ------------------------------------------------------------------ | ------------- | ----------------------------- | +| Does it a usual network traffic based on source and destination IP | PATTERN | Source\_IP -> Destination\_IP | + +## Packet Size + +| Business Check | OwlDQ Feature | | +| ---------------------------------- | ------------- | --------------------------- | +| Is the Packet Size NULL or Missing | BEHAVIOR | AUTO | +| Packet Size within normal range | PATTERN | Source\_IP -> Packet\_SizeB | + + + +## Resulting OwlCheck + +``` +-f file:///home/danielrice/owl/bin/demos/infosec/ -d tab \ +-fullfile -fq "select * from dataset" -encoding UTF-8 -ds infosecv2 \ +-rd "2020-04-04" -dl -dlinc Destination_IP,Packet_SizeB,Source_IP \ +-dlkey Source_IP -fpgon -fpginc Destination_Workgroup -fpgkey Source_Workgroup \ +-df "yyyy-MM-dd" -loglevel INFO -h 10.142.0.29:5432/owltrunk -owluser admin \ +-fpgsupport .000000001 -fpgconfidence 0.4 +``` + +## Which components did we use? + +OwlDQ address the issue of efficient network traffic classification by performing unsupervised anomaly detection and use this information to create dynamic rules that classify huge amounts of Infosec data in real time. + +By providing Infosec dataset along with anomaly records, OwlDQ outlier and pattern algorithms found the anomaly in the network traffic. It mainly detect the following anomalies. + +1. Traffic between Atlanta->Texas +2. The packet size extremely low between Atlanta->Texas +3. Atlanta source IP and Texas Destination IP. + +Realtime OwlDQ can provide the alerts on network traffic anomalies which can help network admins to do further deep analysis and take preventative measure which is daunting task with huge amount of data. + +## Sample Dataset + +{% file src="../../.gitbook/assets/infosec-anomaly.csv" %} + + + +## + diff --git a/projects/use-cases/financial-fxrate-data.md b/projects/use-cases/financial-fxrate-data.md new file mode 100644 index 00000000..088d52e9 --- /dev/null +++ b/projects/use-cases/financial-fxrate-data.md @@ -0,0 +1,49 @@ +--- +description: Foreign Exchange Rates +--- + +# Financial FxRate Data + +### Owl Automatically Alerts to Incorrect FX Rate Data without a Single Rule + +FX Rate data commonly looks like the below table. Often you have a TO currency and a FROM currency with the RATE being the multiplier column for conversion. For example in March you would need to spend $1 US Dollar and 18 US cents to receive $1 Euro. "In exchange for" + +| TO\_CURR | FROM\_CURR | RATE | DATE | +| -------- | ---------- | ----- | ---------- | +| USD | EUR | 0.82 | 2019-03-12 | +| EUR | USD | 1.18 | 2019-03-12 | +| USD | YEN | 111.0 | 2019-03-12 | + +### 28,000 Currency Pairs + +There are roughly 28,000 currency pairs and the exchange rates change throughout the day but at a minimum most banks are concerned with the daily close of the FX Rate. Now imagine trying to write a rule for each currency pair. You'd have to know the relationship and adjust a static threshold for each of the 28K pairs every couple days to keep the rule in tact. Quickly our minds jump to a conclusion that we might be able to solve this with simple math? We can get closer using averages or percent change formulas but these formulas quickly come up short when some currencies commonly fluctuate more than others. Ah I got it (variance), that college class that I thought I'd never use will finally come in handy. Our minds then quickly graduate from stats 101 to 201 and we could consider the individual variance of every combination. But even this will only get us so far as time is an important dimension, the length of time or window can often be tricky to calculate. The problem gets harder when you run your basic stat model and receive a bunch of false positive alerts. You quickly realize that signal-to-noise ratio is important, confidence factors are important, down-training individual foreign currencies that don't seem to fit you statical model are important. Knowing if you copied the data incorrectly, truncated 9 levels of precision on the decimal or if the source provider sent the wrong information is important. Needing the ability to flag exceptions in production on a single currency pair while not flagging the other 27,445 pairs. Using a feedback loop so that the data steward interactions are captured and learned from Vs. having to take the same corrective action over and over. What happens when there is a typo in the currency pair or a single pair goes missing? The answer is that rules don't scale and we need much more than just one off statistical metrics to have a robust and trust worthy DQ program. + +### Consistency + +Even when it is possible to deploy a team of smart guys to build a solution to handle this use-case, the question then becomes but what about all my other data, don't I want similar yet different controls on everything? Especially since FX Rate data by itself doesn't mean that much and is often combined with a number of other datasets to produce value. What if those datasets aren't accurate either? But those datasets have very different columns, different relationships and different time windows. Owl takes an auto-learning approach whereby it interrogates and runs fitness tests against each dataset individually to devise the best statistical and learning approach. The goal being to provide an automated and elegant way to have consistent controls across all your datasets. + +#### Auto Adapting OwlCheck for FxRate Data + +```bash +-ds fx_rate \ +-rd $rd \ +-dl \ +-dlkey TO_CURR,FROM_CURR \ +-q "select * from FX_RATE where date = '${rd}'" \ +-cxn dbConnName \ +-dupe -dupeinc TO_CURR,FROM_CURR -depth 0 +``` + +#### What this OwlCheck Does? + +* Automatic correlation and relationship analysis +* Histograming and Segmentation analysis +* Anomaly detection +* Currency pair tracking +* Schema evolution +* Removes 28K static rules +* Duplicate detection for redundant currency pairs + + + + diff --git a/projects/use-cases/healthcare-dq-in-2-minutes.md b/projects/use-cases/healthcare-dq-in-2-minutes.md new file mode 100644 index 00000000..5a5d5af0 --- /dev/null +++ b/projects/use-cases/healthcare-dq-in-2-minutes.md @@ -0,0 +1,15 @@ +--- +description: >- + OwlDQ connects all members of the healthcare continuum with trustworthy, + timely, and meaningful patient data, while reducing the time, expense, and + effort required by 70% +--- + +# Healthcare Data Quality + +Poor data quality in healthcare is the leading problem that maligns patient outcomes. Hospitals and health information exchanges (HIEs) still struggle with patient matching issues, with most citing data quality problems and poor algorithms as top barriers to patient matching. Correctly linking patient data across organizations is a key element of value-based care, patient safety, and care coordination. Duplicate or mismatched records can result in privacy risks, claim denials, redundant medical tests or procedures, and reporting errors. + +The lack of accurate and reliable DQ in healthcare leads to dire consequences that are completely preventable, as shown in OwlDQ's troponin example below. Complete and accurate data is a vital component of our complex health system, and anything less is an unacceptable risk. OwlDQ provides the predictable data quality that healthcare organizations need to deliver high-quality care that we all strive to achieve. + +![](<../../.gitbook/assets/owldq-healthcare (1).jpg>) + diff --git a/projects/use-cases/insurance-data.md b/projects/use-cases/insurance-data.md new file mode 100644 index 00000000..a63ab88f --- /dev/null +++ b/projects/use-cases/insurance-data.md @@ -0,0 +1,41 @@ +--- +description: >- + Poor DQ is the primary cause for diagnostic providers receiving incomplete + health care payments during Revenue Cycle Management (RCM) +--- + +# Health Insurance Claims Data + +Revenue Cycle Management is the process of identifying, collecting and managing the practice’s revenue from payers based on the services provided. A complete RCM process is critical for a healthcare practice to maintain financial viability and continue to provide quality care for their patients. + +Inaccurate claims data, is the primary cause for diagnostic providers receiving incorrect payments for their services. Most providers struggle with the quality of the data that they receive, and without direct access to the patients, it can be an expensive, laborious process to correct incomplete, or missing data that is required for claim reimbursement. + +Cleaning up or correcting incomplete data is not a step in the claims process that can be skipped. It must be done to assure the reimbursement process is accurate, and complete in the agreed time frame. Automating the data quality during intake is the key to the timely completion of the reimbursement process, and saving the cost and effort of correcting the data down stream. + +``` +Increase revenue from insurance and patient payments +Spend less time tracking down missing patient information +Lower error processing rates +Reduce operating costs +Improve claim processing speed +``` + +The revenue cycle includes all the administrative and clinical functions that contribute to the capture, management and collection of patient service revenue, according to the Healthcare Financial Management Association + +![](<../../.gitbook/assets/RCM Process.png>) + +``` +Preregistration - Collecting preregistration information +Verification - Patient eligibility and benefit is verified +Transcription - recording the diagnoses and procedure +Medical Coding - Properly coding diagnoses and procedures. +Charge capture - Medical services into billable charges. +Claim submission - Submitting claims to insurance companies. +Claim Rejection - when necessary +Payment Posting - Determining patient balances, collection +Secondary Claim Submission +Denial Management - Applying or rejecting payments remittance +Medical Appeals - Examining the necessity of medical services. +Refund - where aplicable +``` + diff --git a/projects/use-cases/intraday-positions.md b/projects/use-cases/intraday-positions.md new file mode 100644 index 00000000..84d25531 --- /dev/null +++ b/projects/use-cases/intraday-positions.md @@ -0,0 +1,96 @@ +# Intraday Positions + +It is common for financial organizations to receive a steady stream of files that have hourly or minutely data. The files might trail the market in a near real-time fashion. Below is an example + +``` +--positions/ + |--2019/ + |--01/ + |--22/ + position_2019_01_22_09.csv + position_2019_01_22_10.csv + position_2019_01_22_11.csv + position_2019_01_22_12.csv +``` + +### File Contents @ 9am + +| TIME | COMPANY | TICK | SIDE | QTY | +| ---------------- | --------- | ---- | ----- | --- | +| 2019-01-22 09:00 | T\&G | xyz | LONG | 300 | +| 2019-01-22 09:00 | Fisher | abc | SHORT | 20 | +| 2019-01-22 09:00 | TradeServ | def | LONG | 120 | + +### File Contents @ 10am + +| TIME | COMPANY | TICK | SIDE | QTY | +| ---------------- | ------- | ---- | ----- | --- | +| 2019-01-22 10:00 | T\&G | xyz | LONG | 280 | +| 2019-01-22 10:00 | BlackTR | ghi | SHORT | 45 | + +Notice that during the day you may or may not have a position for every company recorded. We need a way to link the "company" to its position throughout the day but not alert in cases where they simply did not trade or adjust their position. Owl offers real-time outlier detection for this scenario (see code snippet below). We also need to make sure that each companies position is only represented once per file (per hour in this case) because positions are already the aggregate view of the trades, so they should be unique. Owl offers duplicate detection (see code snippet below). + +### Owl DQ Pipeline + +```scala +// Part of your pipeline includes the ingestion of files that have the date +// and hour encoded in the file name. How do you process those files using Owl? +// +// Format: ___.csv + +val filePath = // positions/2019/01/22/positions_2019-01-22_09.csv + +// Configure Owl. +val opt = new OwlOptions +opt.dataset = "positions" +opt.load.delimiter = "," +opt.load.fileQuery = "select * from dataset" +opt.load.filePath = file.getPath + +opt.outlier.on = true +opt.outlier.key = Array("COMPANY") +opt.outlier.timeBin = TimeBin.HOUR + +opt.dupe.on = true +opt.dupe.include = Array("COMPANY", "TICK") +opt.dupe.exactMatch = true + +// Parse the filename to construct the run date (-rd) that will be passed +// to Owl. +val name = file.getName.split('.').head +val parts = name.split("_") +val date = parts.slice(2, 5).mkString("-") +val hour = parts.takeRight(1).head + +// Must be in format 'yyyy-MM-dd' or 'yyyy-MM-dd HH:mm'. +val rd = s"${date} ${hour}" + +// Tell Owl to process data +opt.runId = rd + +// Create a DataFrame from the file. +val df = OwlUtils.load(opt.load.filePath, opt.load.delimiter, spark) + +// Instantiate an OwlContext with the dataframe and our custom configuration. +val owl = OwlUtils.OwlContext(df, spark, opt) + +// Make sure Owl has catalogued the dataset. +owl.register(opt) + +// Let Owl do the rest! +owl.owlCheck +``` + +### Owl Web + +![](<../../.gitbook/assets/positions-owl (1).png>) + +### DQ Coverage for Position data + +* Schema Evolution +* Profiling +* Correlation Analysis +* Segmentation +* Outlier Detection +* Duplicate Detection +* Pattern Mining diff --git a/projects/use-cases/security-reference-data.md b/projects/use-cases/security-reference-data.md new file mode 100644 index 00000000..dcfcca16 --- /dev/null +++ b/projects/use-cases/security-reference-data.md @@ -0,0 +1,30 @@ +--- +description: Pattern recognition for cross column, categorical, conditional relationships +--- + +# Security Reference Data + +Given the interconnected, automated nature of the data generated by reporting, exchanges, and source systems - hidden patterns go unnoticed. + +Financial firms of all shapes and sizes ingest financial data for a variety of reasons. A few vendors include Bloomberg, Thomson Reuters, ICE Data Services or SIX Financial Information. + + **In no uncertain terms, critical business decisions rely on the accuracy of this data.** + +![](<../../.gitbook/assets/Screen Shot 2019-12-31 at 12.23.52 PM.png>) + +This data is not monolithic and most real-world data easily consists of over 100 columns. Maintaining the quality can be challenging, given the variety of sources feeding into just a single feed. Even the most simple quality checks can snowball into a daunting task. Everything from tickers, sedols, cusips, products, sub-products, issuers, and issuing countries can further complicate the problem. Identifying anomaly values earlier in the data ingestion process can significantly reduce downstream complexity. Furthermore, finding improbable patterns before they're used for making decisions can save costly remediation efforts. + +### An easy way to think about Pattern Analysis. Columns that belong 'together'. + +![Certain zip codes always appear alongside certain states](<../../.gitbook/assets/Screen Shot 2019-12-31 at 12.31.52 PM.png>) + +A common application of OwlDQ is to identify securities that are violating historical patterns. Conditional dependencies can be discovered and used as a guide to highlight misreporting. Rather than defining look-up tables, reference datasets, and predefining exact conditions - security specific patterns can be derived using probabilistic inferences. + +By clicking columns that belong together, a robust framework for cross-column pattern analysis can be generated. The equivalent of dozens of multi-column, conditional rule checks can be applied with just a few clicks. In creating a confidence and probability weighted classification algorithm, this is both adaptive and granular. + +![](<../../.gitbook/assets/Screen Shot 2019-12-31 at 12.32.41 PM.png>) + +Using this technique, OwlDQ is able to reduce false positives, scale coverage, and quickly model a more complex series of checks than domain experts would want to develop. + + + diff --git a/projects/use-cases/smart-meter-data.md b/projects/use-cases/smart-meter-data.md new file mode 100644 index 00000000..ebb797ce --- /dev/null +++ b/projects/use-cases/smart-meter-data.md @@ -0,0 +1,32 @@ +--- +description: >- + OwlDQ uncovered $10 million dollars in unbilled revenue for a leading US + Energy Company. +--- + +# Smart Meter Data + +OwlDQ's Smart Meter Data Analytics provides accurate and predictable data quality to companies often inundated with massive amounts of data and aging enterprise systems. + +The U.S. smart meter data management market forecast is projected to reach $556.94 million by 2026. In 2018, U.S. electric utilities had about 86.8 million advanced (smart) metering installations. OwlDQ provides an automated process to manage the mountain of data collected, and glean critical business insights. By applying our ML algorithms during the normal data ingestion cycles Owl uncovered $10 million dollars in unbilled revenue for a leading US Energy Company. + +In the example below, Owl detected 200 records missing from the previous run. + +![](<../../.gitbook/assets/Public Energy.jpg>) + +Consider the following opportunities that smart meter data analytics provide: + +* Generate new customer insights +* Manage and prevent outages +* Improve maintenance techniques +* Build predictive models for program planning +* Develop new services and rate plans based on customer requirements + +Identification of unbilled revenue: Meter events and usage information can help illustrate a picture of the customer’s energy usage over time. This can help detect energy theft, meter tampering, and equipment issues that may be affecting service levels. + +Outage event analysis and prevention: Today, some utilities are still unable to verify an outage unless personnel physically visit the suspected problem area to confirm. With outage event analysis, however, the utility can know the exact piece of equipment causing a problem, along with the customers directly impacted by it. + +Meter quality assurance: Focusing on meter reading performance enables utilities to ensure reliability. When meter readings are expected but not delivered, the system provides an alert, and calculates overall data score from previous runs . Utilities are notified to potential data quality issues they never would have identified in the past. + +It is meter data analytics that will enable utilities to tackle the problems of the future. + diff --git a/projects/why-owl/README.md b/projects/why-owl/README.md new file mode 100644 index 00000000..2f96cc33 --- /dev/null +++ b/projects/why-owl/README.md @@ -0,0 +1,62 @@ +# Our Approach + +### Because: using raw data to drive key decisions, leads to incorrect answers and end-user distrust. + +_OwlDQ is singularly focused on providing your end-users with the highest standards of data quality. We are purpose-built to solve the problem of data quality and to ensure end-user trust._ + +Whether you use a BI tool to visualize data or you are responsible for serving data to downstream subscribers, you always want to trust that your data is accurate. Showing inaccurate data in a bar chart or PDF report leads to a lack of confidence in the data provider. Take, for example, the data pipeline below. There are 4 main stages: Data Loading, Data Preparation, Data Verification (DQ), and Data Reporting which covers a broad category of all ways to see and deliver data. + +![](<../../.gitbook/assets/Screen Shot 2019-12-18 at 12.55.28 PM.png>) + +In order to avoid getting lost in the latest marketing jargon, a fundamental description is provided under each of the 4 stages. There are many ways to ingest and transform data; the descriptions are not meant to be exhaustive. Imagine a scenario where data is loaded in either a batch or stream, then joined to another dataset with some column transformations, and finally made viewable in a BI tool for consumption. But what about quality? What checks and verifications are in place to guarantee data accuracy and completeness? After all, showing someone a housing report with incorrect estimated housing values or a stock report with the wrong stock prices won’t go over well. Figure 2 below has popular company logos overlaid in each stage to bring more context to the discussion. There are easily 30+ software companies in each of the 4 stages, Owl chose 3 popular companies in each sector at random. Owl is not ranking companies. Gartner is of course an obvious choice if you are looking for companies rankings per sector. + +![](<../../.gitbook/assets/Screen Shot 2019-12-18 at 1.18.39 PM.png>) + +## So, What’s the Problem? + +Detecting data issues is nuanced, manual and time consuming. The traditional solution is to write bespoke code or use a rules engine to validate specific columns in a dataset. If missing data is a concern a common remedy is to write a _nullcheck_. Another common example is a _row count check;_ a piece of logic that checks if the number of rows in a dataset is greater than a specified number. Of course, DQ and business rules can get much more complicated. Scale becomes a huge issue, because it is nearly impossible to write all the rules that a business truly needs to be confident in their data. Often times, the math is _f(x) = columns \* dbTables_. 100 columns on average and 500 tables in a single warehouse equals 50,000 rules if you only wrote 1 rule per column. The reality is you need many rules per column, and your business has more than 500 tables and files. But there are even bigger problems with this strategy. Rules are a reactive approach to solving the problem; they are manually written and they don’t adapt (they are static). With a rules only approach you can measure your franchise risk by the number of rules you can write. This requires coders, domain experts and a tool to write and then manage the rules. + +![](<../../.gitbook/assets/Screen Shot 2019-12-18 at 1.48.13 PM.png>) + +### How Can Predictive DQ Help? + +Owl intentionally solves the problem using a machine learning first, rules second based approach. Owl automatically puts all columns under quality control. This includes _nullchecks, emptychecks, statistical profiles, sketches._ Owl creates snapshots and baselines in order to benchmark past data and discover _drift_. Owl automatically creates a ML labeling system for users to collaborate and down-train items with a click of a button. The reason for this approach is obviously to maximize coverage while reducing the dependency of manual rule building. The greater technical benefit is that all of Owl's generated checks and rules are adaptive. Owl is constantly learning from new data and will make predictions in many cases for: typos, formatting issues, outliers and relationships. This is a paradigm shift **from**, _risk being a measure of how many rules one can dream up and write_, **to** _simply click the Owl \[RUN] button_. __ + +![](<../../.gitbook/assets/Screen Shot 2019-12-18 at 1.50.34 PM.png>) + +## Why a Unified DQ Solution? + +Aren't their other DQ companies and solutions on the market? Yes, absolutely. The challenge is the vast ways IT groups consume and process data. You need to find a product that can plug into Files the same way it can plug into DB Tables, Cloud File Systems, Data Frames and Kafka Topics etc... You need to find a product that offers a consistent feature set and covers all 9 dimensions of DQ. For most companies DQ is an after thought, they will add-on a single dimension of DQ such as _rules_ or _data drift._ Owl offers a full DQ suite to cover the unique challenges of each dataset. Complete coverage and consistency drives trust. A single scoring and reporting framework with 9 pluggable features that can be activated in a tailorable DQ pipeline. Owl is horizontally scaleable, it can scan data from any location with infinity scale. Data quality needs to be abstracted from data ingestion for management to have a single normalized view of data health. + +![](../../.gitbook/assets/owl-unified-dq.jpg) + +## Do One Thing Extremely Well + +Owl believes that data quality is such an important part of the data lifecycle that it requires a company which is solely committed to revolutionizing the way enterprises manage DQ. This is why Owl has a prescriptive approach to DQ (ML first, Rules second). The Owl software is purpose built for predicting and detecting DQ issues. Much like how Jira is used as the standard for software project management even though it is absolutely possible to manage project line items in an excel sheet. Businesses that manage a lot of data require Score Cards, Alerts, Reports, List Views, Collaboration, Down Training, Cataloging, Scheduling and much more. + +## Get Started + +Email us: info@collibra.com + +### Does your DQ Solution Have? + +![](../../.gitbook/assets/OwlDQ-Framework.png) + +| | | +| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **Unified DQ** | The ability to score and manage and report on all datasets (files, tables, topics) agnostically. Providing a single pane of glass for DQ across all data sources. | +| **Collaboration** | The ability for end-users to down-train, annotate and audit each DQ item | +| **Auto Discovery** | The ability to figure out issues in your data without requiring domain experts and rule writers | +| **Anomaly Detection** | The ability to detect numeric and categorical outliers in a dataset | +| **Correlation Analysis** | The ability to measure the lift or relationship between numeric columns | +| **Relationship Analysis** | The ability to discover and make predictions on hidden relationships in your data | +| **Alerting** | The ability to send out alerts when DQ scores drop | +| **Scheduling** | The ability to schedule DQ jobs with a click of a button in the UI | +| **Profiling** | The ability to provide descriptive statistics about the current run overlaid with the past runs for trend analysis | +| **Reconciliation** | The ability to validate the source and target dataset in timeline snapshots | +| **Duplicate Detection** | The ability to find exact and similar matches in data records | +| **Lineage Graphs** | The ability to asses business impact via a business impact score by understanding how connected each dataset is | +| **Schema Evolution** | The ability to detect changes in data types, additions and removals | +| **Rules** | The ability to write custom rules for simple and complex scenarios | + +## diff --git a/projects/why-owl/data-quality.md b/projects/why-owl/data-quality.md new file mode 100644 index 00000000..737b702b --- /dev/null +++ b/projects/why-owl/data-quality.md @@ -0,0 +1,9 @@ +# DQ is the difference + +### Lake vs Swamp + +The difference between a business-critical lake and a swamp is **data **_**quality**_**.** One organization’s Data Lake may be another's Data Swamp. The difference lies in how data is curated. A Data Lake describes a vast amount of data that can be stored, assessed, and analyzed. A Data Swamp has little data governance, DQ automation, or contextual metadata. + +The accuracy and cleanliness of data is directly proportional to the quality of insights end-users will derive. Data lakes that gain broad adoption have strong governance programs. The challenge is, adding a DQ program typically takes 6-12 months but the project never really ends due to the volume, variety and velocity of incoming data. OwlDQ uses autoML so solve this problem. OwlDQ constantly monitors the lake with native integration and unlimited scale. Use OwlDQ to generate the equivalent of 10K rules, while continuously adapting to the natural variance in your data. When erroneous data enters your lake OwlDQ will alert the data steward and provide a rich visual displaying the break records and explainable AI describing the issue. OwlDQ's approach is to learn from data and become incrementally smarter each day to ensure a statistically defensible DQ program. + +### diff --git a/projects/why-owl/our-story.md b/projects/why-owl/our-story.md new file mode 100644 index 00000000..cd4230bc --- /dev/null +++ b/projects/why-owl/our-story.md @@ -0,0 +1,43 @@ +# Our Story + +## Background + +The team comes from a variety of backgrounds. While some spent a decade building technology to detect financial crimes, others were architecting data fabrics at fortune 100 companies. + +> _Regardless of the industry or experience, we all faced similar challenges as it related to data quality._ + +These unique vantage points have allowed us to understand the most common data quality challenges organizations are facing. + +## What Did We Notice? + +We tried many of the traditional tools and techniques. The biggest problem was always the amount of time it took to do everything needed to implement and maintain data quality controls. + +You get left with half-baked data quality coverage and the right controls are added only after issues occur. + +It turned out teams were doing the same tasks for every dataset and for each department, building the exact same tools over and over. + +{% hint style="danger" %} +The result was a never-ending cycle of data issues, fire-drills, and a mad scramble to fix it. Fast. All within the context of real-time business operations. +{% endhint %} + +## Traditional Approach + +Traditional approaches are very manual. + +Start by opening a sample or spreadsheet and conduct analysis (Table-by-table, column-by-column, query-by-query, and item-by-item). + +Next, manually craft the rules, conditions, tolerances, and thresholds. + +Then stitch together the dashboards, scoring, workflows, alerts, and reporting. And you wonder why bare-minimum coverage is common. + +{% hint style="warning" %} +You're only as good as the rules you thought to write. +{% endhint %} + +#### Fast Forward + +Now that the surface area of the data in an organization is so much larger, these traditional techniques don't hold up. + +## What Did We Need? + +What we needed didn't exist. As lifelong data enthusiasts, we wanted a tool that could alert us when data changes occurred without complicated setup and lengthy analysis. We sought something that could work on big, medium, and small data and across all storage formats. Upon evaluating all the commercially available tools, and assessing costs and time of homegrown solutions, there were no great options. diff --git a/projects/why-owl/what-is-owldq.md b/projects/why-owl/what-is-owldq.md new file mode 100644 index 00000000..88e47621 --- /dev/null +++ b/projects/why-owl/what-is-owldq.md @@ -0,0 +1,154 @@ +--- +description: OwlDQ is an intelligent data validation tool. +--- + +# What is OwlDQ + +### Relief for Backlogged Resources. No more bottlenecks for business users. Confidence your data is validated. + +## 8 Ways to Add-Value Using OwlDQ + +1. **Crowdsourcing** + + “People that have never written SQL are now helping with data quality” +2. **Rule Coverage** + + “Did in 20 days what took 2 years with our legacy tool” +3. **Audit & Identify Gaps** + + “Audited our existing checks and could not imagine the gaps we uncovered.” +4. **Automate Repeatable Processes** + + “Owl cut 60% of our manual workloads” +5. **Technology Limitations** + + “We now scan files and Kafka, avoiding downstream issues” +6. **Getting standard** + + “No more piecemeal reports. Files, Warehouse, Lake. All metrics in one, transparent place.” +7. **Building Reports, Visuals, Workflows** + + “This takes the place of 3 tools” +8. **Prioritized Efforts** + + “Easy to see top priorities for improvement” + +{% hint style="danger" %} +* Another expensive project missing deadlines? +* Tired of wasting an afternoon unwinding ETL / ingestion jobs? +* Know the dread of another fire drill? +* Is it crazy to think your time can be better spent than wading through data issues? +{% endhint %} + +### OwlDQ Detects Unintended Data Errors Without Human Bottlenecks. + +{% embed url="https://www.youtube.com/watch?v=sR6ACeO8d34" %} +Systematically Eliminate Your Biggest Data Blind Spots. +{% endembed %} + +### **Focus on Adding Business Value & Avoid Expensive, Complex Commitments** + +## What Savings Does OwlDQ Provide? + +### **Save Hours of Effort with Auto-generated Data Validation Checks** + +{% hint style="success" %} +* **Top 10 Bank** + + Reduced 60% of their manual Data quality workload + $1.7M cost savings + + +* **Top 3 Healthcare Organization** + + Saved 2000 hours during a cloud migration requirement + + +* **Top Insurance Organization** + + Satisfied Regulatory Second Line Controls in a 4-weeks (what took 2 years using their previous tool) + + +{% endhint %} + +### _**While Reducing System-Wide Pain Points**_ + +{% hint style="danger" %} +* Overwhelmed with tickets +* Business users find issues first +* Touchy pipelines break with minor updates +* Too busy responding to fire drills to implement new projects +{% endhint %} + +## How Can OwlDQ Help? + +#### _Click a button and smile - knowing baseline validation checks are applied - instead of spending hours manually digging through data & stitching together scripts_ + +* [x] **Implementing Checks** + + * Autodiscovery + * Generates SQL validations, parameters & thresholds + * Rule suggestions + + +* [x] **Taking Inventory** + + * Bulk Profiling & Metadata Collection + * Data Mapping with Column Identification + * Map Column Fingerprints, Cross-Table Matches & PII Checks + + +* [x] **Consolidating Systems** + + * No more closed-systems or confusing scripts + * Macro & micro views for measuring effectiveness over time + * Global management Across Sources / Platforms / Environments + + +* [x] **Enabling More Users** + * Self-Service, Easy to use Rule Editor + * Pre-Built Analytics and Charts + * Extensible APIs, Open Architecture + +#### **Boost productivity. 80% faster than manual coding. Minimize development costs. Get faster, easier access to data quality metrics. Show line of business users how to self-service.** + +## What **makes OwlDQ unique**? + +### **OwlDQ is The Only Tool Business & Technical Users Will Love** + +Every feature, visual, and component within Owl is intended to make the analysis and implementation of data checks easier. + +![](<../../.gitbook/assets/Screenshot 2020-07-19 at 7.31.37 PM.png>) + +## Why Use Owl? + +### Because Humans Can’t Predict Every Which Way Data Can Go Wrong. + +{% tabs %} +{% tab title="Billing Issue Example" %} +| "An unexpected ETL update occurred during a migration that changed our up-to-date-payments indicator from TRUE/FALSE to 1/0. Needless to say, we were very surprised when invoices were not sent. The rework and reconciliation were super painful. An enormous amount of my time was wasted." | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +{% endtab %} + +{% tab title="Financial Data Example" %} +| "One of our 200+ reference data feeds introduced a pipe (\|) into a position field. The field was defined as VARCHAR so this was allowed. Our models went crazy and we thought we breached risk limits. We ended up selling out of positions (losing millions). Only to uncover the root cause much later that week." | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +{% endtab %} + +{% tab title="API Example" %} +| "We pull data from many APIs. One platform accounts for 10% of enrichment activities (i.e. how we monetize our data). Our auth token accidentally had a daily quota imposed, yet job control said green light (successful connection). We still loaded some rows (1k), just not entire payloads. This was super nuanced. We literally lost \~10% revenue that month." | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +{% endtab %} + +{% tab title="IoT / Meter Example" %} +| "When we introduced new meters, they were hooked up and sending valid readings. They were valid values within valid ranges. Turned out their default setting was rounding the actual values and we were losing precision. Devastating, considering the amount of precision required with blood values." | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +{% endtab %} +{% endtabs %} + +### Schedule a Conversation + +OwlDQ offers a low-hanging fruit opportunity to extend your data quality toolset. + +The fact that data quality is a consistent pain point suggests it's important to many business-critical functions and legacy products not getting the job done. + +### **** diff --git a/release-notes.md b/release-notes.md new file mode 100644 index 00000000..14ee798d --- /dev/null +++ b/release-notes.md @@ -0,0 +1,636 @@ +# Release Notes + +## 2022.03 (Coming soon) + +#### Fixes / Enhancements + +* Rules + * Additional HealthCare Data Classes to Rule Library + +{% hint style="info" %} +Rule Discovery Terminology Alignment + +Data Concepts => Data Categories + +Semantics => Data Classes +{% endhint %} + +* Admin + * Edge download page within Admin Console (for Cloud customers) +* +* Validate Source + * \*Tech Preview\* \[TP] Update Source Scope + +#### Known Limitations + +* Validate Source + * \*Tech Preview\* \[TP] Update Source Scope + * Only works for JDBC connections. Feature is hidden for remote, temp, local files + * Valsrc query won't be updated automatically when modifying column mappings. Use 'Preview' button to reset the feature if column mappings need to be changed + +## 2022.02 + +{% hint style="info" %} +For new Standalone Collibra DQ installations, please double check 'Number of Core(s)' field when setting up 'Edit Agent' +{% endhint %} + +{% hint style="info" %} +Added UUIDs for Jobs may take additional time on initial startup after upgrade +{% endhint %} + +#### Enhancements + +* DQ Job + * Added UUIDs for jobs for better tracking between web and core + * Improved DQ Job page load performance by optimizing calls + * Fixed issue DQ jobs would fail when -rd is in "yyyy-mm-dd HH" format +* Outliers + * [\*Tech Preview\* \[TP\] Outlier Calibration](dq-visuals/more/outliers.md#tech-preview-tp-outlier-calibration) + * Feature flag can be set within owl-env.sh or configMap with `export outlier_calibration_enabled=true` (Default is off) + * Ability to suppress Outlier observations for a user-determined length of time that would have otherwise surfaced as anomalies + * Once feature is enabled, accessible within Outliers tab on DQ Job page +* Alerts + * Ability to navigate to dataset specific Alerts from DQ Job page + * Ability to test SMTP alert configurations when adding an email relay + * Fixed issue where 'Reply Email' field did not properly accept user input value + * Please note there are no (Collibra imposed) domain restrictions on Reply Email field +* Security + * Stricter password policy is enforced on all user/tenant management screens/APIs. + * The restriction is as follows: Minimum length of 8 characters. + * Maximum length of 20 characters. + * At least one upper-case letter. + * At least one numeric character. + * At least one special character (supported are !,%,&,@,#,$,^,\*,?,\_,\~) + * User ID and password cannot be the same. + * Password cannot contain user ID. + * Change Password functionality on user profile requires a current password of the user. + * Mitigated 64 critical, 15 high, and 12 medium vulnerabilities identified by JFrog ([internal-only report link](https://docs.google.com/spreadsheets/d/1yDfqfBO3T3uyw9z\_-zr2SESinTuRWmP\_TgYf\_tlHlk8/edit#gid=0)) + * Upgrade Log4J to 2.17.1 + * Please follow [upgrade steps](release-notes.md#note-to-standalone-collibra-dq-customer-upgrades-we-have-upgraded-to-log4j-2.17-please-refer-to-for) + * Added connection security checks to users to prevent running jobs and query the tables that are not authorized per connection. This is applicable when `DB Connection Security` is enabled in the Admin Console under `General`. + * Implemented stricter session management + * Implemented CORS restriction to mitigate potential CSRF vulnerability + * Enforced strict CORS policy by not allowing any domain. In order to allow other domains and tweak this behavior, we have exposed the following properties as environment variables in owl-env: + * CORS\_ALLOWED\_ORIGINS=[http://facebook.com,http://google.com](http://facebook.com,http/google.com) + * CORS\_ALLOWED\_METHODS=GET,POST,OPTIONS,DELETE,PUT,PATCH + * CORS\_ALLOWED\_HEADERS=X-Requested-With,Origin,Content-Type,Accept,Authorization + * CORS\_EXPOSE\_HEADERS= + * CORS\_ALLOW\_CREDENTIALS=false + * CORS\_MAX\_AGE=0 +* \*Tech Preview\* \[TP] [Collibra Native DQ Connector](integration/dq-connector.md) + * Fixed issue where tenant specified on DQ Connector configuration (issuer of the jwt token field within DGC Edge Management page) was not properly accepted; only rules that existed with 'public' schema were brought over; now the DQ Connector will accept the proper values +* Agent + * Upon potential deletion of an agent, added server side validation to indicate number of scheduled jobs so that users can understand if jobs fail going forward +* Rules + * Enhanced stability on Parallel Rule execution to ensure all rules load by reverting back to fixed thread counts + * Display exceptions upon rule execution failure to improve rule management experience + * Improvements to user experience in Rule Library tab (within Rules page) including filters and column alignment + * Quick Rule dropdown within the Rules page will save with default severity of 1 point and a threshold of 1 percent + * Enhanced validation for rules generated in Profile tab + * Fixed issue where removing semantic tag may not have removed corresponding auto-generated rule + * Rule name character limit of 100 + * Rule Builder page now returns error messages where the dataset contained 0 records +* Catalog + * Renaming Dataset from Catalog page keeps associated rules + * Clone only creates the dataset shell (with DQ job run configs, no additional rules, etc.) will be copied + * Bulk actions support for Data Concepts + * Fixed issue where child of business unit could be assigned as parent + * Fixed issue where clearing individual filters were not functioning +* Validate Source + * \*Tech Preview\* \[TP] New collapsible section for Query in Source tab; enables users to use custom srcq, similar to query on section on Home tab so that users do not need to edit -srcq in cmd line editor on Run tab + * Introducing new observation types via `-valscrshowmissingkey` flag + * Key not in source + * Key not in target + * Source Name should be fetched as part of getcatalogandconnsrcnamebydataset API call for a given dataset + * Fixed issue which prevented Hive from working as Target +* Export / Import + * Fixed issue that import could not accommodate more than one table insert + * Fixed bug where certain values were inadvertently inserted into RegEx rules upon Export +* Connection + * Fixed Out Of Memory issue with Dremio + * Explicitly added limit clause in the preview query within Update Scope + * Dremio driver requires double quotes in Schema, Table, and Column names e.g. "SchemaName"."TableName" + * Fixed Oracle TIMESTAMPLTZ conversion error +* Explorer + * Fixed issue where 'Analyze Table' option did not populate for Hive + * Fixed the static date values showing up in Managed Template and Run Check while running the job via v2/runtemplate API call from swagger UI +* Files + * File names with spaces are now handled with double quotes t + * Implemented Supported File Type Check at time of uploading the Temp Files via Explorer + * Default supported file types are “csv,json,parquet,avro,delta". + * In order to add/update the supported file types and ensure validation, a new environment variable needs to be added in owl-env.sh as below: `export ALLOWED_UPLOAD_FILE_TYPES="csv,json,parquet,avro,delta"` + * Tip: For remote files with delimiter, please use the csv dropdown options for files with .txt extension + * \*Tech Preview\* \[TP] Users have ability to assign an agent when using temp file and local file Explorer paths without manually appending -master to agent or job (previous known limitation) + * LIMIT values are now properly accepted on the Scope & Range query panel +* Dupes + * Fixed issue where column selections were not retained from the original DQ Job with Dupes ON for future runs + +#### Known Limitations + +* Rules + * Cannot currently create rule with API /v3/rules; will be fixed in future release + * Please use /v2/createrule API +* Profile + * Stat Rules + * Tool tips will only generate when Max Precision and Max Scale are greater than 0 +* DQ Job + * /v2/runtemplate API still creates 'zombie' job + * Please use /v3/jobs/run +* LinkID + * LinkID column selection is case sensitive; breaks may not appear if case does not match +* Outliers + * Outlier Calibrate + * Outliers cannot retrain on-demand; to suppress existing Outliers, must rerun the DQ Job for those date(s) + * In-app labels do not exist for Outliers which have been subject to past, current, or future calibration; references only exist within the `outlier_boundary` table in the metastore + +**\[Informational Only] New Tables Introduced To Metastore In 2022.02** + +* `outlier_boundary` + +**\[Informational Only] Changes To Metastore Made In 2022.02** + +``` +ALTER TABLE validate_source_metadata ADD COLUMN IF NOT EXISTS validate_values_show_missing_keys boolean DEFAULT false +ALTER TABLE opt_source ADD COLUMN IF NOT EXISTS validate_values_show_missing_keys boolean DEFAULT false + +ALTER TABLE opt_source ADD COLUMN IF NOT EXISTS filter_cols character varying[] + +ALTER TABLE user_profile ADD COLUMN IF NOT EXISTS external_user_id VARCHAR + +ALTER TABLE owlcheck_q ADD COLUMN IF NOT EXISTS agent_job_uuid UUID +ALTER TABLE job_log ADD COLUMN IF NOT EXISTS job_uuid UUID +ALTER TABLE platform_logs ADD COLUMN IF NOT EXISTS job_uuid UUID +ALTER TABLE platform_logs DROP CONSTRAINT IF EXISTS platform_logs_job_uuid_ux +ALTER TABLE platform_logs ADD CONSTRAINT platform_logs_job_uuid_ux UNIQUE (job_uuid) +ALTER TABLE opt_owl ADD COLUMN IF NOT EXISTS job_uuid UUID +``` + +## 2022.01 + +#### Enhancements + +* DQ Job + * Fixed issue where backrun "-br" flag was inadvertently added on future runs (error contained in 2021.12) if the initial DQ Job setup Explorer selected backrun + * Improved validation to not allow for slashes in dataset name +* Validate Source + * Fixed potential DQ Job failure with Source turned on for some legacy installations when upgrading from older versions to 2021.11 and newer +* Explorer + * `DB_VIEWS_ON` can be added with TRUE or FALSE values by adding new App Config (Add Custom within Admin -> Configuration) + * \-Addlib flag now working across JDBC connections + * Update Scope now supports rdEnd +* Rules + * When creating rules, run-time limit for each rule (in minutes) can be set on the Rule page UI and on the V3 API (by setting `runTimeLimit` property). The default is 30 minutes if not explicitly set. This 30 minute limit sets the overall timeout limit for all rules in a particular job. For example, if there are 10 rules with 9 rules with 30 min limit and 1 rule as 90 min limit, then the DQ Job will wait up to 90 min for all 10 rules to finish. This is because all rules must finish before the Rule stage in DQ Job to finish and move to the next stage. We do not support async stages where one long running rule is running while the job itself moves on to the next stage. + * Added ability to specify score of 0 to a rule + * Improvement to Stat Rules to fail without exception when result is not within range +* Profile + * Fixed ability to remove a business unit from a dataset + * Fixed issue where data concepts were not correctly displaying on a dataset's Profile page + * Fixed sensitive labels not being assigned from Discovery + * Treat certain doubles, floats, decimal types as Decimal format that preserves length and prevents Java from truncating to E11 format + * Removed commas when displaying date columns +* Security + * SAML Login fix for IDPs that use POST binding as default +* S3 + * Enhanced support where "." in column headers were causing large jobs to not complete + * Underscores now replace periods and large jobs should no longer hang +* Connections + * Updated default Snowflake template connection properties + * Corrected 'db' parameter placeholder on connection string versus former 'databaseName' + * Added [BigQuery connection troubleshooting](connecting-to-dbs-in-owl-web/supported-drivers/connectivity-to-bigquery.md#views) information + +#### Known Limitations + +* Local files using NO\_AGENT require a valid $SPARK\_HOME on the machine where the web server is running. +* Supported data types + * CLOB datatypes are unsupported +* Explorer + * \-Addlib not yet supported for Remote Files e.g. S3 + +#### \[Informational Only] Changes To Metastore Made In 2022.01 + +``` +ALTER TABLE owl_rule ADD COLUMN IF NOT EXISTS run_time_limit DOUBLE PRECISION NOT NULL DEFAULT 30.0; +ALTER TABLE owl_rule ADD COLUMN IF NOT EXISTS scoring_scheme INT4 NOT NULL DEFAULT 0; + +ALTER TABLE job_log ALTER COLUMN stage TYPE character varying; -- stage set to varchar because RULE logs rule_nm into stage +ALTER TABLE job_log ALTER COLUMN log_desc TYPE character varying; +ALTER TABLE job_log ALTER COLUMN log_hint TYPE character varying; +``` + +## 2021.12 + +#### \*Note to Standalone Collibra DQ Customer Upgrades\*: We have upgraded to Log4J 2.17, please refer to [standalone-upgrade.md](installation/standalone/standalone-upgrade.md "mention") for additional steps + +#### Enhancements + +* Rules + * Semantic and data concept management: Run Discovery feature + * Run Discovery feature can be accessed from Catalog by selecting 'Data Concept' option from Actions or clicking the 'Run Discovery' button on the Rules tab of the DQ Job page. This will run a DQ Scan to detect for the semantics assigned to the selected data concept + * Algorithm now selects best match if column matches 2 or more data classes based on % match and name distance + * \*Tech Preview\* \[TP] Configurable rule break preview limit + * Global default is 6 max rows per rule + * Any change from 6 must be specified with previewLimit (API /v2/createrule) or in the Preview Limit field (UI) + * Maximum of 50 from UI + * Introducing additional Stat Rules including minPrecision, maxPrecision, minScale, maxScale +* Behavior + * Min and max value checks are now triggered for all numeric columns when selected, even if column contains zeroes in lookback period + * AR column view graph now shows theMean value for current day (runId). No re-run of DQ Job is necessary. The displayed Mean makes it clear that the % change is the % change from the mean, not runId - 1 day. + * Findings in behaviors that were directly correlated to a row count shift as the root cause have been optimized, such that a major deviation in row count will no longer down-score related fields in the dataset to reduce noise +* Catalog + * Catalog now features intelligent ranking based on Recency, Most Scanned, User +* Outliers + * [Dynamic minimum history](dq-visuals/more/outliers.md#dynamic-history-options) allows for gaps in dates when establishing lookback period, which is established by history with row count > x (specified by user) + * Fixed issue where outlier data preview graphics were not displayed + * Fixed issue where outlier results did not honor the initial scope where clause, in particular for Remote Files (S3) +* Connections + * BigQuery: Enhanced support for cataloging host name +* Pulse View + * Pulse view can filter Connections and Users + * Pulse view can serve as proxy verification on whether scheduled jobs were successfully completed +* Profile + * Viewable precision and scale statistics for double, float, and decimal data types +* Shapes + * Fixed issue where data shape preview not available when same shape is detected on the same row for different columns +* Files + * \*Tech Preview\* \[TP] Users have ability to assign an agent when using temp file and local file Explorer paths + * Known limitation: -master must be freeform appended to the agent or to each job + * Support for multicharacter delimiters + * Improved delimiter support to distinguish string commas versus actual CSV commas to align data to respective columns +* Agent + * Fixed issue where certain completed jobs could not be re-run on the DQ Jobs page. In other words, NO\_AGENT was the only available option in the Agent dropdown. Now, users can select valid agents in the dropdown and this will persist for future scheduled jobs +* Schedule + * Implemented validation to enforce user to choose days when picking schedule to avoid Java error messages +* Explorer + * Fixed issue where '&' was not properly supported when adding additional parameters +* API + * JSESSIONID session time is configurable + * Bearer token and JSESSIONID authentication paths are properly forked +* Pattern + * Patterns activity now shows Count (number of times the current dataset has the Pattern breaks). This Count is interpreted the same way as Outlier activity Count + +## 2021.11 + +#### Enhancements + +* Rules + * \*Tech Preview\* \[TP] [Semantics and data concepts management](dq-visuals/rules/data-concepts-and-semantics.md) + * The application now supports dynamic semantics checks. This allows you to create custom semantics that can be checked for when running a DQ check on a data set. Previously the application checked against a predefined set of semantics. You also have access to controls to organize and apply these semantics checks. The following is a list of changes: + * There is a new data concepts management page. You can access it from Catalog or Admin Console. You can assign multiple semantics to a data concept. + * When running a DQ check, you can select a data concept. The semantics assigned to this data concept will be checked against each column of dataset. + * You have a list of predefined semantics that are not editable. You also have the ability to create/edit/delete custom semantics. + * Repo on rules page has been added to Rules Library where semantics will be viewable. +* Resource Limits + * You can edit the [Performance Settings](benchmarks/performance-settings.md) to supply limits to executors, cores, memory and cells so that a user can be warned if submitting a job that requires a lot of resources and admins can control maximum resources submitted. + +#### Enhancements + +* Explorer + * \*Tech Preview\* \[TP] Dynamic query reload allows you to view JOIN query columns in other activities. + * User can update and reload the schema table with the custom query in the scope section by clicking the \[Update Scope] button. It will enable using the new columns from the custom query in all activities (Profile, Outlier, Dupes, Patterns, Source) + * Since the first tab is for compositing the query, updating fields will change the user's custom query. Therefore, all areas are locked except the "query" field in the first tab to keep the query unchanged after updating the scope table + * Support for some special characters in table name. + * Fixed the ability to add additional libs that were previously not being properly saved on subsequent runs. Under DQ Job tag, please utilize -fllb boolean (union lookback) and libsrc input box for lib directory path (will materialize as -addlib). +* Connections + * \*Tech Preview\* \[TP] BigQuery Views and Joins + * Please add the following to the BigQuery connection property + +``` +viewsEnabled=true +``` + +* API + * You can perform multiple imports without conflicts. + * You can have an incremental import such as updating matching records / insert new / leave existing. There is no requirement to delete tables first before running import. +* Profile + * Fixed backrun timebin to work with weeks and quarters instead of days. +* Outliers + * Split historical load to avoid historical query rounding up. + * \*Tech Preview\* \[TP] [Dynamic minimum history](dq-visuals/more/outliers.md#dynamic-history-options). +* Source + * Fixed an issue where settings were not sticky for subsequent runs. +* Security + * SAML Enhancements + * New [configuration settings](security/configuration/authentication-with-saml-idp/#set-the-saml-authentication-properties) are available when the Load Balancer is set for SSL Termination. + * You can now set the [**RelayState** property](security/configuration/authentication-with-saml-idp/multi-tenancy-support-through-saml-relaystate.md) to route SSO to the proper tenant. + +#### Patches + +* 2021.11.1 Explorer + * Allow ampersand in metastore host name for additional parameters + * In below example, support for ampersand needed for required SSL flags + +``` +metastore01.us-east1-b.c.customer-dq-prod.internal:5432/dev?sslmode=required¤tSchema=public +``` + +#### Known Limitations + +* Rules + * Semantics and data concepts: + * Not supported in pushdown mode + * Exporting RegEx semantics not currently supported + * While it is possible to create joins and cross-dataset rules using Freeform SQL, it is best practice to create a view and handle the join prior to running the DQ Job. +* Behavior + * Schema is not eligible for invalidate +* Files + * Local files using UPLOAD\_PATH, UPLOAD\_FILE\_PATH, and temp files are only eligible to be deployed using the default NO\_AGENT option. These are only intended for quick tests and not intended for production-scale use. Best practice is to use a remote file system connection (S3, Google storage or ADLS). + * Delimiter support for special characters is limited. Supported file delimiters are comma, pipe, tab, semicolon, double quote and single quote. Custom delimiters will work for many characters, but not all combinations. + * Temp files and NO\_AGENT should have -master local\[\*] or -master spark://:7077 defined in freeform append of the agent options +* DQ Job + * When submitting jobs via API from a different machine with a different timezone, timezone discrepancies are not accounted for automatically. Best practice is to align each component to use UTC. + * Jobs submitted via API with a run date that include HH:MM in the -rd (run date) will submit to the job queue and leave a remnant ‘STAGED’ job +* Connections + * Postgres limits max connections per spark job. The default is 100. Please refer to Postgres official documentation how to increase max\_connection and shared\_buffers. + * https://www.postgresql.org/docs/9.6/runtime-config-connection.html + * BigQuery + * Updating scope to include joins in BigQuery can only be materialized when tables are part of the same dataset collection + * Should you receive an error for pre-existing BigQuery jobs, please add -dssafeoff to the cmd line or select ‘Allow Overwrite’ to enable this from Edit mode in the Explorer +* Alerts + * After an upgrade to 2021.11, you may need to set the environment variable `ALERT_SCHEDULE_ENABLED=true` in **owl-env.sh** and restart **owl-web** to enable email alerts to work again. + +## 2021.10 + +#### Enhancements + +* DQ Job + * Refactored DQ Job Score to Gauge Chart +* Explorer + * Fixed issue where permissions are checked on datasets that do not yet exist +* Connections + * Sybase 'Test / Preview' now available + * Updated web model of saving additional connection properties + * Fixed scenario where editing connection yields null instead of empty for multiple values +* Rules + * Placeholder new searchable Rule Summary Page for Rule statistics / insights +* Alerts + * Updated Alert Mailer to TLS 1.2 to resolve Third Party Error exception + * Fixed issue where alerts are deleted even when clicking cancel button +* Behavior + * Fixed issue where user must refresh to have invalidated item removed from UI +* Search + * Fixed search on Audit Datasets and Dataset Management page +* Scorecards + * Date ranges are now customizable +* Validate Source + * Added feature that provides 'trim' option on String columns when running source-target validation, extra spaces in the cell are trimmed on both ends (left and right) +* Dupes + * Resolved issue with white spaces in column headers blocking duplicate detection +* Security + * Added configuration for setting the SAML\_ENTITY\_BASEURL, which sets the Consumer service url for the SP Metadata +* Shapes + * Fixed issue where custom values override even after toggling Shapes back to auto or off +* Console + * Fixed uncaught TypeError on login screen + * Fixed GET timeout error on registration page +* Export/Import API + * Users will be able to run the export/import API calls to conduct multiple promotions on the repo, schedule, and rule tables. + +#### Patches + +* 2021.10.1 Import / Export API without constraint conflicts + * Import must match exactly to the format of our export in order to parse out columns and values to perform an update when existing records are already there + +``` +owl_rule +owl_check_repo +job_schedule +rule_repo +``` + +#### Known Limitations + +* File sizes + * Individual files greater than 5gb will experience performance degradation in Explorer for Standalone installs. Best practice is to save in smaller chunks and use bypass schema in the Explorer if needed. + * Individual files greater than 25gb will experience performance degradation in Core for Standalone installs. +* Files + * Explorer / browser will generally have difficulty supporting > 250 columns in files +* Profiling + * Pushdown profiling on Bigquery, Redshift, Athena and Presto is available for specific datatypes. + * Backrun option and flag will persist beyond the first run (-br). Please remove this flag if you do not want to backrun again. +* Explorer + * QUARTER and WEEK are not supported time bins in this release. + * On non-csv files, Explorer will not automatically infer file types. Users must change file type to the required value and click Step 2 "Load File". Nothing will change in Step 1 "File Information". A future enhancement will be added to automatically check filetypes by reading the first file + * Dataset names should not contain special characters +* Rules + * Out of the box semantic rules cannot be edited (STATECHECK, GENDERCHECK, etc). Users can still apply their own global rules which can be customized. + * LinkId does not support alias columns that are not part of the -LinkId definition +* Connections + * Connection names should not contain spaces +* Validate Source + * Complex Validate Source queries can only be edited from the CMD line or JSON directly before hitting Run. +* Security + * Active Directory in Azure SQL can connect via LDAP (basic auth) or Kerberos. +* S3 / GS / ADLS + * Remote storage connections should be defined using the root bucket only. +* Estimate Job is only available for files when Livy is being used. +* Stop Job on jobs page is limited and does not work for all installation types. +* Bigquery connector does not work with views + +## 2021.09 (09-2021) + +#### New Feature + +* Alert + * Alert notification page displaying a searchable list of alerts emails sent. [Email Alerts](workflows/email-alerts.md#example-alert-in-your-inbox) +* Job Page + * UI refresh + * New chart with successful and failed jobs + +#### Enhancements + +* Profile + * When faced with a few errors e.g. 0.005% null, highlight issues more clearly and visibly instead of the notion of rounding up to and displaying 100.0% +* Jobs + * Enhanced query and file date templating and variable options. This allows easier scheduling and programmatic templating for common date variables + * Job Template corrupt time portion of ${rd} on last run of replay + * Refactor job actions column +* Catalog + * Completeness report refactor / consolidation to improve performance +* Export + * Outlier tab in DQ Job page (hoot page) displays linkIds and included in the export +* Security + * Added property for authentication age to reduce token expiration + * UI labels more generic when configuring a connection with password manager script +* Agent + * Agent no longer shows as red if services are correctly running +* Logging + * Jobs log retention policy now configurable in Admin Console -> App Config via "JOB\_LOG\_RETENTION\_HR" (variable must be added along with value). If not added, default to 72 hours + * Platform logs retention policy now configurable in Admin Console -> App Config via "PLATFORM\_LOG\_RETENTION\_HR" (variable must be added along with value). If not added, default to 24 hours +* Outliers + * Fixed connection properties behavior given how multiple custom properties are handled in Hive + * Fixed outliers issue that ignored WHERE clause on remote files +* Scorecards + * Fixed missing search results issue in list view for Patterns type +* Connections + * New templates for Redshift and Solr +* Connections Security + * Ticket Granting Ticket (TGT) authentication for HDFS & Hive + * You can now choose the TGT auth model for connections and point to a TGT file as an additional kerberos authentication model + * Kerberos Principal + Password Manager for Hive + * You can now use a password manager script to fetch a hive password for a princiapl to authenticate + * S3 SAML Auth (TP) + * DQ is configured to use SAML based authentication to S3 buckets with password manager or provided credentials. Testing is limited to OneLogin for SAML Provider in this tech preview release + +**Patches** + +* 2021.09.1 Validate Source +* 2021.09.2 Validate Source Large DB load +* 2021.09.3 Save on datashapes from new DQ Job + +## 2021.08 (**08-2021)** + +_Please note updated Collibra release name methodology_ + +#### Enhancements + +* Explorer + * Support for handling large tables + * Implemented pagination function for navigation + * Improved error handling for unsupported java data types + * Fix preview for uploaded temp files +* Collibra DQ V3 REST APIs + * Additional rest APIs for easier programmatic job initiation, job response status, and job result decisioning (used in pipelines). Streamlined documentation and user inputs allow users to choose any language for their orchestration wrapper (python, c#, java, etc). [More info on Collibra DQ Rest APIs](https://docs.owl-analytics.com/rest-api/apis) +* Patterns + * Fix load query generation issue when WHERE clause is specified +* Behaviors + * Fix behavior score calculation after suppressing AR + * Fix percent change calculations in behavior AR after retrain + * Mean Value Drift \[New Feature] [Behaviors](dq-visuals/behaviors.md#using-behavioral-analytics-change-detection) +* Security + * Introduce new role of ROLE\_DATA\_GOVERNANCE\_MANAGER with ability to manage (create / update / delete) Business Units and Data Concepts. [More info on Collibra DQ Security Roles](https://docs.owl-analytics.com/v/2021.08/security/owl-security/role-based-access-control-rbac) + * Relaxed field requirements for password manager connections for App ID, Safe, and Password Manager Name +* Scorecard + * Enhanced page loading speeds for scorecard pages +* Rules + * Rule activity is now more efficiently parallelized across available resources +* Validate Source + * Pie chart will highlight clearly visible ‘issue’ wedge for anything less than 100% +* UI/UX + * Updated with more distinct icon set + +## 2021.07 (**07-09-2021)** + +_Please note updated Collibra release name methodology_ + +#### Enhancements + +* \*Tech Preview\* \[TP] [Collibra Native DQ Connector](integration/dq-connector.md) + * No-code, native integration between Collibra DQ and Collibra Catalog +* UX/UX + * Full redesign of web user experience and standardization to match Collibra layout + * Search any dataset from any page +* Hoot + * Rules Display with \[more] links to better use the page space + * Auditing for changes per scan +* Explorer + * JDBC Filter enablement by just search input +* Profile + * Add more support for data-concepts from UI or future release +* Behaviors + * Down-training per issue type + * AR user feedback loop (pass/fail) for learning phase +* Scheduler + * [Time based cleanup routine](https://docs.owl-analytics.com/data-retention/time-based-data-retention) +* Security + * SQL View data by role vs just Admin +* Reports + * OTB completeness reports from reports section. [Completeness Report](reports/built-in/completeness-report.md) + +## 2.15.0 (**05-31-2021)** + +#### Enhancements + +* Hoot + * Down-training per activity vs globally +* Logging + * Expose server logs on the jobs page from the agent and cluster +* Explorer + * Enhanced Experience for display of stats for database tables + * Validation for Dupes section to ensure all input is validated before save + * Support for edit mode with Dremio connections + * Allow file scan skip-N to skip a number of rows if extra headers are present in the file + * Support Livy sessions over SSL for files +* Profile + * Add quick click rules based on profile distribution stats +* Behaviors + * Down-training per issue type +* Scheduler + * Support for $rdEnd in the template + * Auto update schedule template based on last successful run + * Support S3 custom config values in scheduled template +* Security + * SAML Auth + * Support for JWT Authentication to the Multi-Tenant management section +* Multi-Tenant + * Support for an alternate display name for each tenant to be displayed in the UI and login tenant selection + +## 2.14.0 (**03-31-2021)** + +#### Enhancements + +* Hoot + * Edit mode on Password Manager supported connections + * Edit mode on complex query + * Behavior chart display of last 2 runs +* Explorer + * ValSrc auto Save + * Remote File + * Support for Google Cloud Storage (GCS) + * Support for Google Big Query + * Folder Scans in Val Src + * Auto generate ds name + * FullFile for S3 + * {rd} in file path naming + * Estimate Jobs (Only on K8s) + * Analyze Days (Only on K8s) + * Preview Data (Only on K8s) +* Connection + * Store source name to connect a column to its source db/table/schema + * Custom driver props for remote file connection +* Profile + * Filtergrams for Password Manager connections + * Filtergrams for Alternate agent path connections + * Filtergrams on S3/GCS data source (Only on K8s) +* Rules + * UX on page size options +* Scheduler + * Support multiple db timezone vs dataset timezone +* Outliers + * Notebook API returns true physical null value in DataFrame instead of string "null" +* Shapes + * Expanded options for numeric/alpha + * Expanded options for length on alphanumerics + +## 2.13.0 (**12-20-2020)** + +#### Enhancements + +* Schema + * Notify of quality issue on special characters in column names +* Shapes + * Shape Top-N display in profile +* Behaviors + * Chart enhancements including visible Min/Max ranges on AR drill in + * Force pass/fail at AR item level +* Explorer + * Menu driven selections on existing datasets + * Run remote file scans at parent folder level +* Scheduler + * Allow scheduling based on dataset timezone (previously all UTC) +* Profile + * Enhanced Drill in display including new AR view & Shape Top-N values + * Data Preview Filtergram Export of distinct column values +* Validate Source + * Additional UI parameters exposed: File Type/ File Delimiter + * Edit support in Explorer Wizard step + * Grouped display on Hoot Page with aggregate statistics +* Business Units + * Organize datasets by [business-units](https://docs.owl-analytics.com/observation-assignments/business-units) visible in catalog and profile views +* Hoot Page + * Ability to Clone datasets from a given dataset run +* Rules Page + * Allow Vertical Resize +* Catalog + * Searchable filters for: rows, columns, scans, and more +* Performance + * 2X faster on data loading activity based on revised caching and surrogate I diff --git a/reports/built-in/README.md b/reports/built-in/README.md new file mode 100644 index 00000000..f6784fae --- /dev/null +++ b/reports/built-in/README.md @@ -0,0 +1,5 @@ +# Built-In + +Collibra DQ provides various out of the box reports allow users to access insights quickly. + +![](<../../.gitbook/assets/image (148).png>) diff --git a/reports/built-in/completeness-report.md b/reports/built-in/completeness-report.md new file mode 100644 index 00000000..c5fd827f --- /dev/null +++ b/reports/built-in/completeness-report.md @@ -0,0 +1,35 @@ +# Completeness Report + +![](../../.gitbook/assets/completeness\_report.gif) + +{% hint style="info" %} +V +{% endhint %} + +Completeness commonly means what percentage of your data is filled in, as in not `EMPTY` or `NULL`. The confusion often comes when you consider the context of completeness. Completeness of a column? Completeness of a dataset (file or table)? Completeness of a collection of tables or business unit? And even worse... completeness over time. + +Fortunately the Collibra DQ Completeness Report covers all of these dimensions, allowing the user to generate a report that answers almost any completeness question. We have also found that volume weighted completeness is arguably a better metric than average completeness in a rollup scenario. + +### Report Builder + +The "all" view shows completeness of the entire DQ program. Really interesting to see that across hundreds of datasets and thousands of total columns or data assets we are 92% complete. The more accurate measurement is likely the 97% due to the fact that it represents the volume of data as well. + +![](<../../.gitbook/assets/Screen Shot 2021-08-12 at 5.26.43 PM.png>) + +When looking at Completeness over time you may need to differentiate between the time when the DQ job ran (update time) or the date/time the data represents (run date). For example I load stock data today but the data I loaded was for last week. + +### Column Completeness vs Dataset Completeness + +![](<../../.gitbook/assets/Screen Shot 2021-08-12 at 4.04.53 PM.png>) + +Above you can see the dataset is 34.7% complete this is made up of columns ranging from 0% to 100% complete. This view makes it easy to see which columns are the least complete and likely the biggest DQ concern. + +### Why is Completeness a concern for DQ? + +Data completeness is a large concern because it means the data is missing all together. Data can be inaccurate, invalid, of the wrong type but missing leaves you without any data. The challenge is that sometimes data values can be missing for good reasons, this leaves the challenge up to the business user to understand the context of whether missing data is bad for business or acceptable. Please read about Collibra DQ [Behaviors](../../dq-visuals/behaviors.md) detection including `NULL` and `EMPTY` detection for a more modern approach to completeness management. In this approach we will use the data itself to create baselines and profiles to understand which completeness issues actually matter and should be actioned. + +### Why Behavioral Analytics for Completeness is Better? + +In many academic settings completeness is defined for cases where you expect the data to be complete. However in the real world you have too much data and never enough time to document what and when data should be complete. It is vastly more efficient to use Collibra DQ `Replay` and `Profile` features to generate _statistical process control_ around completeness. This will alert you to a change in slope, or in other words a drastic change in completeness, which tends to be the exact DQ events you care the most about. + +![](<../../.gitbook/assets/Statistical Process Control for Data Quality (Animation) Aug 2021.gif>) diff --git a/reports/built-in/coverage-report.md b/reports/built-in/coverage-report.md new file mode 100644 index 00000000..d9f540d4 --- /dev/null +++ b/reports/built-in/coverage-report.md @@ -0,0 +1,13 @@ +# Coverage Report + +## How to Generate a Data Quality Coverage Report + +In the explorer view where you see the file tree explorer for each database you can click the pie chart \[coverage report]. This will show all the datasets that have been cataloged and that have DQ jobs running. The time-series bars shows the run level information aggregated by month, while the donut chart shows the percent coverage of the entire database or schema. + +{% embed url="https://www.youtube.com/watch?v=LIJDoIN82W8" %} + +### File tree explorer Generate Report + +22 out of 44 tables in public schema have been DQ jobs running on them. Likely have profiles and auto rules for validation. 24 out of 44 have been cataloged, which means they are registered and have metadata. + +![](<../../.gitbook/assets/Screen Shot 2021-04-03 at 7.02.23 PM.png>) diff --git a/reports/built-in/owl-summary-reports.md b/reports/built-in/owl-summary-reports.md new file mode 100644 index 00000000..688e684c --- /dev/null +++ b/reports/built-in/owl-summary-reports.md @@ -0,0 +1,9 @@ +# Summary Reports + +![](../../.gitbook/assets/reports.gif) + +### Owl Weekly Rollup Reports + +When operating a large data lake or several large data environments it can be helpful to be able to report across different dimensions at an executive summary level. You may wish to know the health or coverage per line of business, department or tenant or per database. A single snapshot in time usually doesn't tell the whole story. Owl automatically aggregates a simple series of the high level trends for each dataset. This allows a business user to see the DQ scores and trends as well as row counts and passing runs at a glance. These reports can be exported by clicking the file type you prefer in the header. + +![](<../../.gitbook/assets/Screen Shot 2019-08-27 at 10.57.30 PM.png>) diff --git a/reports/built-in/profile.md b/reports/built-in/profile.md new file mode 100644 index 00000000..0cc9d4e3 --- /dev/null +++ b/reports/built-in/profile.md @@ -0,0 +1,19 @@ +# Dataset Report + +### Owl Dataset Report + +Search for a particular dataset and run to generate an exportable dataset profile report. Click on the report icon\* in the left navigation and choose Dataset report. Provide a valid dataset name in the auto-complete search box to see the report output. + +### Navigate to the Profile Report Section + +![](<../../.gitbook/assets/image (100).png>) + +### Search for a dataset + +![](<../../.gitbook/assets/image (129).png>) + + + +### View or Export the results + +![](<../../.gitbook/assets/Screen Shot 2020-05-07 at 6.53.09 PM.png>) diff --git a/reports/custom.md b/reports/custom.md new file mode 100644 index 00000000..8ffa7468 --- /dev/null +++ b/reports/custom.md @@ -0,0 +1,111 @@ +--- +description: BI Integrations +--- + +# Custom + +Custom reports can be leveraged by connecting your favorite BI tool on the underlying reporting mart. Below are a few queries that can be used as inspiration for building your own reports. Please refer to the[ ERD diagram ](../architecture/diagram/erd.md)for a larger list of tables. + +**Long Running Jobs** + +`select dataset,run_id,total_time from dataset_activity where total_time is not null order by total_time desc` + +**Jobs Submitted** + +`select * from owlcheck_q` + +**Jobs by User** + +`select count(*) as owlchecks, username from owlcheck_q where updt_ts < now() group by username order by owlchecks desc` + +**Jobs by User, Dataset** + +`select count(*), user_nm, dataset from dev.public.owl_check_history group by user_nm, dataset order by count desc` + +**Largest by Row Count** + +`select dataset,rc as row_count from dataset_scan order by rc desc` + +**Jobs by Month** + +`with grp as ( select date_trunc('MONTH', run_id) as by_month from dataset_scan where run_id < now() ) select count(*) as owlchecks, by_month from grp group by by_month order by by_month desc` + +**Rules by User** + +`select count(*) as rules, user_nm from owl_rule group by user_nm order by rules desc` + +**By Spark(Cluster) Usage** + +`select * from opt_spark order by num_executors desc` + +**Jobs IDs from Agent** + +`select remote_job_id from agent_q where remote_job_id is not null` + +**Dataset Activity** + +`select dataset,run_id,total_time from dataset_activity where total_time is not null order by total_time desc` + +**Jobs with Enriched Metrics** + +`with activity as ( select dataset,run_id,total_time from dataset_activity where total_time is not null order by total_time desc limit 100), scans as ( select * from dataset_scan where dataset in (select dataset from activity) ), configs as ( select * from opt_spark where dataset in (select dataset from activity)), schema as ( select count(*) as col_cnt, dataset from dataset_schema where dataset in (select dataset from activity) group by dataset ) SELECT A.dataset, A.run_id, C.total_time, A.rc, D.col_cnt, B.driver_memory, B.num_executors,B.executor_cores, B.executor_memory, B.master FROM scans A INNER JOIN configs B ON A.dataset = B.dataset INNER JOIN activity C ON A.dataset = C.dataset and A.run_id = C.run_id INNER JOIN schema D on A.dataset = D.dataset ORDER BY C.total_time desc` + +**Jobs. Load Times and Resources** + +`with activity as ( select dataset,run_id,total_time from public.dataset_activity where total_time is not null order by total_time), scans as ( select * from public.dataset_scan where dataset in (select dataset from activity) ), configs as ( select * from public.opt_spark where dataset in (select dataset from activity)), schema as ( select count(*) as col_cnt, dataset from public.dataset_schema where dataset in (select dataset from activity) group by dataset ) SELECT A.dataset, A.run_id, A.updt_ts, C.total_time, A.rc, D.col_cnt, B.driver_memory, B.num_executors,B.executor_cores, B.executor_memory, B.master FROM scans A INNER JOIN configs B ON A.dataset = B.dataset INNER JOIN activity C ON A.dataset = C.dataset and A.run_id = C.run_id INNER JOIN schema D on A.dataset = D.dataset ORDER BY A.updt_ts desc limit 10` + +**Dataset Scans and Scores By Schema** + +`select * from public.dataset_scan where dataset like 'public.%';` + +**Dataset Scans and Scores By Name** + +`select * from public.dataset_scan where dataset ='public.atm_customer';` + +**Scans By Month By Schema - 'Public'** + +`select dataset, DATE_TRUNC('MONTH', run_id) as run_id, count(*) as Total_Scans from dataset_scan where dataset like 'public%' group by dataset, run_id order by run_id asc` + +**Rule Breaks Past 30 Days** + +`select * from rule_output where run_id < NOW() - INTERVAL '30 DAY';` + +**Scheduled Jobs Queue** + +`select job_id,agent_id,dataset,run_id,status,activity,start_time from public.owlcheck_q;` + +**Column Counts from Dataset Schema** + +`select dataset, count(*) from dataset_schema group by dataset;` + +**Profiling Stats** + +`select dataset, run_id, field_nm, (null_ratio * 100) as null_percent, (empty_ratio * 100) as empty_percent, ROUND( CAST( ( 100 - ((null_ratio * 100) + (empty_ratio * 100)) ) as numeric), 3) as completeness from public.dataset_field where updt_ts > '2020-06-01' and dataset = 'ProcessOrder' and run_id > '2021-03-17 00:00:00+00' order by completeness desc` + +**Metadata / Schema / Datatypes** + +`select * from public.dataset_schema;` + +**Profile Stats** + +`select * from public.dataset_field;` + +**Locate Similar Columns** + +`select distinct dataset, field_nm, max_abs from dataset_field where max_abs = 'Wireless Telecommunications'` + +**Same Column Names** + +`select distinct dataset, field_nm from dataset_field where field_nm = 'authenticated_user'` + +**Similar Column Names** + + `select distinct dataset,field_nm from dataset_field where field_nm like '%id%'` + +**Behavior Findings** + + `select * from behavior where dataset='esg_data'` + +**All Columns for Schema from Postgres Stats** + +`SELECT table_name FROM information_schema.tables WHERE table_schema = 'public' ORDER BY table_name` diff --git a/scheduler/schedule-management/README.md b/scheduler/schedule-management/README.md new file mode 100644 index 00000000..98272b86 --- /dev/null +++ b/scheduler/schedule-management/README.md @@ -0,0 +1,15 @@ +# Schedule Management + +![](../../.gitbook/assets/job\_schedule.gif) + +Enable Scheduled Jobs from your environment variable in owl-env.sh + +SCHEDULE\_ENABLED = TRUE/FALSE (Default = TRUE) + +Limit Scheduler Open Time-slots: + +If you don't want automated jobs to be running during business hours, or for a particular day/time on any given day of the week you can set "off-limit" times so authorized users don't select them when scheduling a job. + +![](<../../.gitbook/assets/image (43).png>) + +![](<../../.gitbook/assets/image (44).png>) diff --git a/scheduler/schedule-management/view-re-run-scheduled-jobs.md b/scheduler/schedule-management/view-re-run-scheduled-jobs.md new file mode 100644 index 00000000..a69e8458 --- /dev/null +++ b/scheduler/schedule-management/view-re-run-scheduled-jobs.md @@ -0,0 +1,6 @@ +# View/Re-Run Scheduled Jobs + +You can view your schedule jobs from the scheduled tab on the jobs page + +![](<../../.gitbook/assets/Screen Shot 2020-07-27 at 8.37.09 PM.png>) + diff --git a/scheduler/schedule-owlchecks.md b/scheduler/schedule-owlchecks.md new file mode 100644 index 00000000..35d2c0f4 --- /dev/null +++ b/scheduler/schedule-owlchecks.md @@ -0,0 +1,18 @@ +# Schedule a Job + +After you successfully run a job, you can schedule that job to run automatically. You can do this by updating the template (if needed) and clicking the schedule icon in the hoot page. To change the template, you can use the -rd variable: ${rd} in your query to set dynamic dates or date ranges for your scheduled job. + +![](../.gitbook/assets/schedule.gif) + +![](<../.gitbook/assets/image (45).png>) + +![](<../.gitbook/assets/Screen Shot 2020-07-27 at 8.32.20 PM.png>) + +Here you can choose the Agent to run the job, the frequency (daily/monthly/quarterly) and the time of day: + +![](<../.gitbook/assets/Screen Shot 2020-07-27 at 8.28.09 PM.png>) + +If your monthly or quarterly jobs are loaded after the month or quarter has ended, you can schedule the job for the day when the data has landed, but set the offset to the proper run date required for charting/reporting. + +![](<../.gitbook/assets/Screen Shot 2020-07-27 at 8.30.57 PM.png>) + diff --git a/scorecards/dataset-scorecard/README.md b/scorecards/dataset-scorecard/README.md new file mode 100644 index 00000000..a72cd6b7 --- /dev/null +++ b/scorecards/dataset-scorecard/README.md @@ -0,0 +1,15 @@ +# Overview + +![](<../../.gitbook/assets/scorecards (1).gif>) + +![](../../.gitbook/assets/scorecards2.gif) + +Scorecards allow you to visualize the health and consistency of a dataset over time. Owl will highlight macro and micro trends, e.g weekend loads vs weekday loads or behavioral item changes per day, and display them on the dataset scorecard. + +![](../../.gitbook/assets/owl-trend.png) + +## DQ Over Time, Drill-In and Roll-Up + +Data quality isn't a one time check or once a year project. Data is the life blood flowing through your organization. It's mandatory to know how your data is behaving right now, yesterday and over time to gain an understanding of the trends. For insights to be meaningful we need to see both the lowest granularity and the big picture. Owl's approach lets you drill all the way into the exact moment the issue arose as well as zoom out to see how your data is behaving month to month. This makes Owl useful at many different levels in your organization's heirarchy- a Data Steward might be more concerned with a recent change in data and want to correct it using service now immediately, whereas a Chief Data Officer might be more concerned with the health of the organization's data overall. + +{% embed url="https://youtu.be/Lu2I7n_nRlc" %} diff --git a/scorecards/dataset-scorecard/owl-dq-screen-shots.md b/scorecards/dataset-scorecard/owl-dq-screen-shots.md new file mode 100644 index 00000000..493fcadb --- /dev/null +++ b/scorecards/dataset-scorecard/owl-dq-screen-shots.md @@ -0,0 +1,19 @@ +# Scoring + +Scoring can be completely controlled by the end-user with out of the box defaults. + +![](../../.gitbook/assets/scoring.gif) + +Owl provides a data quality assessment that scans 9 dimensions of a data set to assure the integrity of that data. The 9 dimensions are behavior, rules, outliers, pattern, source, record, schema, duplicates, and shapes. + +OwlCheck produces a data quality score from 0-100. 100 represents that there were no integrity issues found in the data set. The score numerically represents the integrity of that data. For example, the score of 100 would tell the data analyst that zero data quality issues in that data set. + +Owl will scan your data with the same frequency, that you load your data - Owl scans 9 dimensions of DQ and summarizes the results into a score from 0-100. + +#### Aggregate Score + +![The score starts with 100 and individual dimensions deduct from the total.](<../../.gitbook/assets/image (110).png>) + +{% hint style="info" %} +Each dimension can be custom weighted and rules can contain custom scoring severity. In this example, the deducted score (59) from the starting score (100) equals an overall score of 41. +{% endhint %} diff --git a/scorecards/group-scorecard.md b/scorecards/group-scorecard.md new file mode 100644 index 00000000..6a2ca019 --- /dev/null +++ b/scorecards/group-scorecard.md @@ -0,0 +1,9 @@ +# Page View + +![](../.gitbook/assets/page\_view.gif) + +## Visually and logically group datasets together to create a heat-map of blindspots. + +Similarly to job control and build frameworks like Jenkins, we always want to know the health of our datasets. Did it recently fail? Does it commonly fail on Mondays? What is the aggregate or composite score for multiple datasets? Owl allows you to define scheduled health checks that depend on the success/failure status of any number of datasets. This protects the downstream process consumers from pulling erroneous data into their models. + +![](../.gitbook/assets/owl-group-scorecard.png) diff --git a/scorecards/list-view.md b/scorecards/list-view.md new file mode 100644 index 00000000..831b9f1c --- /dev/null +++ b/scorecards/list-view.md @@ -0,0 +1,11 @@ +# List View + +![](../.gitbook/assets/list\_view.gif) + +Find DQ issues across all datasets in your lake. Ranked them, sort them, search them and limit them by time. + +![](../.gitbook/assets/owl-list-view.png) + +### How Do I Take Action on Lots of Alerts? + +One of the most frequently asked questions is how do I operationalize and take action on all of the issues that are seemingly valid yet overwhelming. Owl seems to find many valid issues in my datasets but there are more issues in my data than I expected. The list view helps by first limiting to a time range, let's say issues that have occurred less than 5 days, or possibly even just issues that occurred today. This will likely be a large reduction in the issue count. Secondly a user might limit issue to DQ issues that have a Business impact. This means other downstream processes or datasets are connected to this dataset and field. You only get the business impact feature if you have enabled the Owl Graph module. This will be another drastic reduction in issues because now we are limited to issues that recently occurred and have impact to the business. Finally we might filter by the "class" or "type" of DQ issue such as Rules or Outliers. It is common in a large data lake that after taking these steps you are left with the 1 or 5 top ranking issues in your lake. These are likely the issues that should be prioritized and moved into a remediation queue. diff --git a/security/configuration/README.md b/security/configuration/README.md new file mode 100644 index 00000000..94c18d7e --- /dev/null +++ b/security/configuration/README.md @@ -0,0 +1,2 @@ +# Configuration + diff --git a/security/configuration/authentication-with-active-directory-ldap/README.md b/security/configuration/authentication-with-active-directory-ldap/README.md new file mode 100644 index 00000000..0aa3df14 --- /dev/null +++ b/security/configuration/authentication-with-active-directory-ldap/README.md @@ -0,0 +1,39 @@ +# Active Directory LDAP + +Go to the Admin Console page (click the Owl -> click the Gear -> click Admin Console). + +Click on “AD Setup” and you will see the screenshot below. + +![](<../../../.gitbook/assets/image (93).png>) + +1. AD Enabled = flag to enabled AD (after binding please restart the owl-web application) +2. Page Size = 100 is recommended. The number of results in a single page (NOTE: For really large AD Environments it’s best to narrow down the Base Path and/or possibly using Group Search Path to narrow down to that group explicitly). +3. Host = Host: ldap://x.x.x.x or ldaps://x.x.x.x Port: is usually 389 for ldap or 636 for ldaps +4. Base Path is the domain specified in the example above owl.com (equals to DC=owl,DC=com). +5. Group Search Path = helps to narrow down list to an explicit group or parent group (example: CN=owladmins) +6. Bind user = \@\ +7. Bind Password = users domain password. +8. Click save and you should receive the message below on the top of the owl-web application. + +![](https://lh4.googleusercontent.com/Z\_btfJeipsC7WQrC2lC80Z9IwmomiBX8VFaNneAgdGOBPRfyArWao7f\_\_C9TEFVXDb0-DyxFpXc3BUrpmhJs20gelNfA8TI7-sVTkyD4aVlV7Q1WUR50dN7MvukyrcBoUysfYgvm) + +When binding to AD you do not need a special AD username and password. The application just needs a user to bind with in order to run a read-only query on the groups. The AD credentials are not stored, owl uses this dynamically to understand what groups you want to map. + +Now the binding is complete we can map an AD Group to an Owl ROLE. + +![](<../../../.gitbook/assets/image (92).png>) + +1. Click on “ROLE MAPPING” +2. Select a ROLE in the drop down list - (Alternatively this is a time you can add a new Owl ROLE to map the AD Group(s) you want to include in that Owl ROLE). +3. Click Load Groups +4. Select Group in the left box (use the filter on top to filter base on what you provide in the text box). +5. click the name to move it left to right. +6. click “Save” and you’ll get a response back to Owl-web like the following. + +![](https://lh5.googleusercontent.com/b6FG3k6y73mbVt9eXl8AG9CORfKRGwvcJhR5pRNtx5F4lkjeWc8ZB6uKSd6M0BpoNmYv6Iw8Aai78XNH4fq3bEe6eITdr5f9DFOy9eBDg5b58KWMf94OZoza8I8cwNPMA3uStoUQ) + +Now that we’ve mapped an AD Group to an AD Role we can now log out and try to login as a domain user. (REMEMBER: you will have to restart the owl-web by running ./owlmanage.sh restart\_owlweb when toggling AD Enabled). + +When logging into the Owl Web application please make sure to append the domain to the end of the user name as depicted below. + +![](<../../../.gitbook/assets/image (91).png>) diff --git a/security/configuration/authentication-with-active-directory-ldap/ad-group-to-owl-role-mapping.md b/security/configuration/authentication-with-active-directory-ldap/ad-group-to-owl-role-mapping.md new file mode 100644 index 00000000..465d670c --- /dev/null +++ b/security/configuration/authentication-with-active-directory-ldap/ad-group-to-owl-role-mapping.md @@ -0,0 +1,31 @@ +# AD Group to Owl Role Mapping + +By Mapping an AD Group to an Owl role you are granting all users from the selected AD Group, role based access to the selected Owl role as depicted in the steps below. There will be more on creating custom application roles in the RBAC Section of this document. + +Application properties set in owl-env.sh can be set to determine which LDAP properties correspond to LDAP Query results. For group mapping you will need the full path (unique) as well as the display name. + +For example: + +``` +LDAP_GROUP_RESULT_DN_ATTRIBUTE=distinguishedname +LDAP_GROUP_RESULT_NAME_ATTRIBUTE=CN +``` + + + +![](https://lh4.googleusercontent.com/jrTaHJvax02-T0eoYKWigqTiB6nzcrPZPNRyI6wmg0pwIQ5Y8w9ZSne1GwMEx7Adtj1jdB8koDdcfniYx7cKQcoCjgi5tQ22yhcKvRlU3Xa9kOxA-KrwBfzM1IafIzyE4Bmdm1NX) + +1. Click on “ROLE MAPPING” +2. Select a ROLE in the drop down list - (Alternatively this is a time you can add a new Owl ROLE to map the AD Group(s) you want to include in that Owl ROLE). +3. Click Load Groups +4. Select Group in the left box (use the filter on top to filter base on what you provide in the text box). +5. click the name to move it left to right. +6. click “Save” and you’ll get a response back to Owl-web like the following. + +![](https://lh5.googleusercontent.com/b6FG3k6y73mbVt9eXl8AG9CORfKRGwvcJhR5pRNtx5F4lkjeWc8ZB6uKSd6M0BpoNmYv6Iw8Aai78XNH4fq3bEe6eITdr5f9DFOy9eBDg5b58KWMf94OZoza8I8cwNPMA3uStoUQ) + +Now that we’ve mapped an AD Group to an AD Role we can now log out and try to login as a domain user. (REMEMBER: you will have to restart the owl-web by running ./owlmanage.sh restart\_owlweb when toggling AD Enabled). + +When logging into the Owl Web application please make sure to append the domain to the end of the user name as depicted below. + +![](http://18.204.201.140:8080/xwiki/bin/download/Documentation/Authentication%20and%20Authorization/WebHome/Screen%20Shot%202019-05-22%20at%2011.13.40%20AM.png?width=291\&height=241) diff --git a/security/configuration/authentication-with-local-user-store/README.md b/security/configuration/authentication-with-local-user-store/README.md new file mode 100644 index 00000000..83daf9a1 --- /dev/null +++ b/security/configuration/authentication-with-local-user-store/README.md @@ -0,0 +1,3 @@ +# Local User Store Authentication + +Local user store authentication is enabled by default, and the product ships with a default user with admin privileges. If you should need to toggle this mechanism on/off you can find the setting by navigating to the Admin Console and clicking on the Security Icon. The setting labeled "Local User Store Enabled" when checked will allow your company to create and administer users stored in Owl's internal database user store. Toggling this feature will require a restart of the web application. diff --git a/security/configuration/authentication-with-local-user-store/adding-local-users.md b/security/configuration/authentication-with-local-user-store/adding-local-users.md new file mode 100644 index 00000000..1265a494 --- /dev/null +++ b/security/configuration/authentication-with-local-user-store/adding-local-users.md @@ -0,0 +1,13 @@ +# Adding Local Users + +Users can request access to the application from the login page by clicking the register link under the login button. Application administrators can also create user accounts by navigating to the Admin Console and clicking on the Users Icon. In either case, simply fill out the account form and a user record will be created. The user account at this stage will not have access to the application until an application administrator grants that user access to an owl role or multiple roles. + +![](http://18.204.201.140:8080/xwiki/bin/download/Documentation/Authentication%20and%20Authorization/WebHome/Screen%20Shot%202019-05-22%20at%2011.32.45%20AM.png) + +![](http://18.204.201.140:8080/xwiki/bin/download/Documentation/Authentication%20and%20Authorization/WebHome/Screen%20Shot%202019-05-22%20at%2011.34.03%20AM.png) + +### Granting Local User Roles + +Application administrators can manage user accounts and role access by navigating to the Admin Console and clicking on the Users Icon. On that page locate the user you would like to modify and click on their username in the table of users. This will load a pop-up window that allows you to modify the user account including Enabled/Disabled as well as the roles they belong to. Click on the Roles tab to begin granting roles to the selected user. The available roles established will be listed on the left, click a role from the left-hand side to apply that role to the user. You can also add all roles by clicking the double arrow icon above the list. When you are satisfied with the role listings on the right-hand side for that user, click the Update button. There will be more on creating custom application roles in the RBAC Section of this document. + +![](http://18.204.201.140:8080/xwiki/bin/download/Documentation/Authentication%20and%20Authorization/WebHome/Screen%20Shot%202019-05-22%20at%2011.34.42%20AM.png) diff --git a/security/configuration/authentication-with-saml-idp/README.md b/security/configuration/authentication-with-saml-idp/README.md new file mode 100644 index 00000000..6d55a3c9 --- /dev/null +++ b/security/configuration/authentication-with-saml-idp/README.md @@ -0,0 +1,127 @@ +# SAML Authentication + +You can integrate Collibra DQ with an existing SAML solution and have your application act as a service provider. Once you set up the environment variables, you can access and configure SAML security settings as an administrator in the **SAML Setup** section of the **Admin Console**. + +### Set the SAML authentication properties + +Before configuring SAML authentication, you must add the following required properties to your configuration + +{% tabs %} +{% tab title="Standalone installation" %} +1. Add the properties as environment variables to your **owl-env.sh** file located in **\/owl/config/**. +2. Prefix all properties with the `export` statement. +3. Restart the web app. +{% endtab %} + +{% tab title="Cloud native installation" %} +1. Add the properties as environment variables to your **owl-web ConfigMap**. +2. Recycle the pod. +{% endtab %} +{% endtabs %} + +#### Required properties + +| Property | Description | +| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| SAML\_ENABLED |

Whether Collibra DQ uses SAML.

If set to false, users sign in with a username and password.

If set to true, SAML handles the authentication request.

| +| SAML\_ENTITY\_ID |

The name of the application for the identity provider, for example Collibra DQ.

It is an immutable unique identifier of the service provider for the identity provider (IDP).

| + +#### Optional properties: general + +You can further configure your SAML setup with the following optional properties. + +| Property | Description | +| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| SAML\_ENTITY\_BASEURL |

The base URL that is provided in the service provider metadata.

Set this property when you use DNS.

| +| SAML\_LB\_EXISTS |

Whether the application needs to configure a load balancer.

You generally need this setting only when the Load Balancer is set for SSL Termination.

The default value is false.

If set to true, you must also provide a value for SAML_LB_SERVER_NAME.

| +| SAML\_METADATA\_USE\_URL |

Whether Collibra DQ uses an URL or a file for the identity provider metadata.

The default value is true.

If set to false, the file must be accessible to the owl-web and the path provided in the Meta-Data URL field of the Meta Data Configurations section under Admin Console --> SAML Setup --> Connection.

| +| SAML\_ROLES\_PROP\_NAME |

The attribute in which the identity provider stores the role of the user authenticating in the SAML response.

The default value is memberOf.

| +| SAML\_GRANT\_ALL\_PUBLIC |

Whether any user authenticated by the identity provider is allowed to login the Collibra DQ application.

The default value is false.

| +| SAML\_USER\_NAME\_PROP | The name of the attribute in the SAML response that contains the username of the user who is authenticating. | +| SAML\_TENANT\_PROP\_NAME |

If using multi-tenant mode, the variable in which the identity provider stores the tenant name of the user authenticating in the SAML response.

The app will attempt to use the RelayState parameter to identify the tenant and then fall back on this property.

| +| SAML\_KEYSTORE\_FILE |

The path to the keystore for SSL validation.

The store should contain the keypair of the identity provider for SSL verification.

| +| SAML\_KEYSTORE\_PASS | The password for the keystore provided in `SAML_KEYSTORE_FILE`. | +| SAML\_KEYSTORE\_ALIAS | The alias of the keypair (private and public) in the keystore used for SSL verification. | + +#### Optional p + +When **SAML\_METADATA\_USE\_URL** is set to `true` (default), the following additional properties are available. + +| Property | Description | +| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| SAML\_METADATA\_TRUST\_CHECK |

Whether to enable Collibra DQ to do trust verification of the identity provider.

The default value is false.

| +| SAML\_METADATA\_REQUIRE\_SIGNATURE |

Whether Collibra DQ signs authentication requests to the identity provider.

The default value is false.

| +| SAML\_INCLUDE\_DISCOVERY\_EXTENSION |

Whether to enable Collibra DQ to indicate in the SAML metadata that it’s able to consume responses from an IDP Discovery Service.

The default value is false.

| + +#### Optional Properties: Load Balancer + +When **SAML\_LB\_EXISTS** is set to `true`, the following additional properties are available. + +| Property | Description | +| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| SAML\_LB\_INCLUDE\_PORT\_IN\_REQUEST |

Whether to include the port number in the request.

The default value is false.

| +| SAML\_LB\_PORT |

The port number of the load balancer.

The default value is 443.

| +| SAML\_LB\_SCHEME |

The protocol of the load balancer.

The default value is https.

| +| SAML\_LB\_SERVER\_NAME |

The server or DNS name.

Usually, the same as SAML_ENTITY_BASEURL without specifying the protocol, for example without https://.

This property is required and has no default.

| +| SAML\_LB\_CONTEXT\_PATH | Any path that may be defined on the load balancer. | + +{% code title="Example" %} +``` +#enable SAML & show the SAML SSO option on the login page +SAML_ENABLED=true + +#set SSL communication properties for SAML +SAML_KEYSTORE_FILE=/keystore.p12 +SAML_KEYSTORE_PASS=**** +SAML_KEYSTORE_ALIAS=**** + +#in multi-tenant mode set the name of the IDP variable to hold the tenat name +SAML_TENANT_PROP_NAME=tenant + +#set the name of the IDP variable to hold the user roles in the response +SAML_ROLES_PROP_NAME=memberOf + +#allow login if authenticated to the IDP +SAML_GRANT_ALL_PUBLIC=true + +#set the EntityId of the application to be supplied to the IDP +SAML_ENTITY_ID=OwlOneLogin + +#optinally set a property that contains the username in the response +SAML_USER_NAME_PROP="" + +#optionally use a file for the IDP metadata vs a URL (default is true) +SAML_METADATA_USE_URL=false + +#optional security settings to +SAML_METADATA_TRUST_CHECK=false +SAML_METADATA_REQUIRE_SIGNATURE=false +SAML_INCLUDE_DISCOVERY_EXTENSION=false +``` +{% endcode %} + +### Download service provider metadata for the IDP + +Once you have enabled and configured SAML authentication, you can download the service provider metadata that is required by your identity provider from `https:///saml/metadata`. + +### Enable the SAML sign in option + +When you are ready with your IDP settings, add the final configuration settings in the **Admin Console**: + +1. Sign in as an existing administrator with a username and password to the tenant you want to configure. +2. In the **Admin Console**, click **SAML Setup**. +3. In the **Connection** tab, select the **SAML Enabled** check-box. +4. In the **Meta Data Configurations** section, click **+Add**. +5. Enter the required information. + +| Option | Description | +| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Meta-Data URL | The URL of the identity provider metadata XML file or the location of the downloaded XML file, depending on how you configured the SAML\_METADATA\_USE\_URL property. | +| Meta-Data Label | The name for this specific configuration. | +| IDP URL | The URL of the Collibra DQ application that is provisioned by the identity provider. | + +Save your changes. + +{% hint style="success" %} +Once you complete this setup, you can restart your application and sign in using the SAML SSO option. +{% endhint %} diff --git a/security/configuration/authentication-with-saml-idp/multi-tenancy-support-through-saml-relaystate.md b/security/configuration/authentication-with-saml-idp/multi-tenancy-support-through-saml-relaystate.md new file mode 100644 index 00000000..061bb350 --- /dev/null +++ b/security/configuration/authentication-with-saml-idp/multi-tenancy-support-through-saml-relaystate.md @@ -0,0 +1,7 @@ +# Multi-tenancy support through SAML RelayState + +Starting with Collibra DQ version 2021.11, in a multi-tenant environment, you can help route SSO to the proper tenant with the SAML provided **RelayState** property. + +When set, the property is sent to the IDP and then returned to the consumer service, such as /saml/SSO. The application checks that value to ensure the correct tenant is set up. + +You can set the **RelayState** property in the in the **SAML Setup** section of the **Admin Console**. diff --git a/security/configuration/connection-security.md b/security/configuration/connection-security.md new file mode 100644 index 00000000..caf9e541 --- /dev/null +++ b/security/configuration/connection-security.md @@ -0,0 +1,17 @@ +# Connection Security + +Log into the Admin Console Page as an Administrator of Owl. + +Go to the Admin Console page (click the Owl -> click the Gear -> click Admin Console). + +Click on “Security” and toggle on “DB Connection Security”. + +Log into the Admin Console Page as an Administrator of Owl. + +Go to the Admin Console page (click the Owl -> click the Gear -> click Admin Console). + +Click on “DB Access” as shown below + +![](https://lh6.googleusercontent.com/evhM3O7A\_MZGhGUrSWXlwRH-CMl1-VC9EJoC\_W1irjUquQSg\_3LeuMFFUAFZ5lWHesQo-6Ulq7wMdM9B-lWzE6lYZEqEl5T47WXG\_mNKpBCi4V5gpsSAPT3C2xNkrBbRO9dv07\_t) + +Here you will see all the established connections. Click on a connection name to begin mapping roles to the connection. Any user in those mapped roles will be able to see the connection in the UI. diff --git a/security/configuration/dataset-security/README.md b/security/configuration/dataset-security/README.md new file mode 100644 index 00000000..6380fcd1 --- /dev/null +++ b/security/configuration/dataset-security/README.md @@ -0,0 +1,23 @@ +# Dataset Security + +Log into the Admin Console Page as an Administrator of Owl. + +Go to the Admin Console page (click the Owl -> click the Gear -> click Admin Console). + +Click on “Security” as described in the “Security Settings” section of this document (just above) and toggle on “Dataset Security”. + +Log into the Admin Console Page as an Administrator of Owl. + +Go to the Admin Console page (click the Owl -> click the Gear -> click Admin Console). + +Click on “Datasets” as shown below + +![](<../../../.gitbook/assets/Screen Shot 2019-09-05 at 11.07.43 PM.png>) + +Note all the Datasets from the demo script we launch at the beginning of this document have been added to ROLE\_PUBLIC ROLE defined in Owl except the row\_count Dataset. At this point we already have the [odemo@owl.com](mailto:odemo@owl.com) user created and mapped to the ROLE\_PUBLIC ROLE which has access to those Datasets. Lets login as [odemo@owl.com](mailto:odemo@owl.com) and try to access row\_count DatasetStats page (for an understanding on how to access the DatasetStats page see section entitled “Understanding the DQ DatasetStats Page” in this document…..doing so will result in the error message below. + +![](<../../../.gitbook/assets/Screen Shot 2019-09-05 at 11.08.43 PM.png>) + +However I can see the other DatasetStats that are part of the PUBLIC\_ROLE as [odemo@owl.com](mailto:odemo@owl.com) is a member of that ROLE. + +The last thing to notice is that as the user [odemo@owl.com](mailto:odemo@owl.com) cannot access the Admin pages the AD Group odemo is a part of the ADMIN\_ROLE. diff --git a/security/configuration/dataset-security/dataset-masking.md b/security/configuration/dataset-security/dataset-masking.md new file mode 100644 index 00000000..d6099ea2 --- /dev/null +++ b/security/configuration/dataset-security/dataset-masking.md @@ -0,0 +1,20 @@ +--- +description: Add column level masking for sensitive data in a dataset +--- + +# Dataset Masking + +## Masking From the UI + +After an owlcheck has been run you can perform column level masking from the hoot page if you have ROLE\_ADMIN or ROLE\_DATASET\_MANAGER. + +{% hint style="info" %} +Masking updates will take effect for all existing and future runs of the selected dataset. +{% endhint %} + +{% hint style="info" %} +ROLE\_ADMIN and ROLE\_DATASET\_MANAGER also have the ability to unmask. +{% endhint %} + +![](<../../../.gitbook/assets/Screen Shot 2019-09-05 at 10.21.44 PM.png>) + diff --git a/security/configuration/role-based-access-control-rbac.md b/security/configuration/role-based-access-control-rbac.md new file mode 100644 index 00000000..0aaa9ffb --- /dev/null +++ b/security/configuration/role-based-access-control-rbac.md @@ -0,0 +1,34 @@ +# Role Based Access Control (RBAC) + +High-level view of Owl’s Security architecture depicted below. + +![](https://lh3.googleusercontent.com/Fw3N1MBaOdHcOrXlOGr9sYn4RGbnakMorQjeLnEJcoihKhaJaZ-H4OtxiXo6b6B6O404EfgpXhegs6OJtUJkurjwJVwSa-haXkPAY5W7tvV0QTN3Y4Zk2qQDZkAn3WWoE2v2S\_ig) + +Whether leveraging a Local User Store, Active Directory, or using the out of the box user accounts that come with Owl via LDIF, security stays the same. An admin can create many ROLE’s. A user, whether local user, LDIF user, or AD user can be part of one or many roles. And a ROLE maps to a Dataset within Owl. + +A unique feature within Owl is the fact that Owl does not store information about external user accounts. This avoids the need to sync external users from an external user store such as AD to owl. Instead Owl will map the external group to an internal role. From here the ROLE can be mapped to the different functionality within Owl whether they are Admins / Users / and have access to different datasets and future functionality. The other benefit is that if a specific userid within the external user store is terminated, when the user is purged from the external user store such as AD they will immediately not have access to Owl’s web application. This is because when the user logs into Owl’s web application that is backed by AD their login will interrogate AD to authenticate the user account. See logical flow below for how the group to role mappings work. + +![](https://lh5.googleusercontent.com/6lYry5CMj2FBQC8mvyrG\_30FvI573q3\_NMm11DHL05UC-5SgH5NRUydAm9qNa-CihLCgA\_e4\_-NEUOqJfgGQgmioIO6QXOhkH8p4s4rACl6EkV7m1tg1ICNlij077p2mBc6qgPKd) + +### RBAC Usages + +Owl supports RBAC configuration with both core roles and custom roles. Core roles include the following: + +| Role | Access Description | +| ---------------------------- | --------------------------------------------------------------------------------- | +| ROLE ADMIN | Modify any access, config settings, connections, role delegation | +| ROLE DATA GOVERNANCE MANAGER | Ability to manage (create / update / delete) Business Units and Data Concepts | +| ROLE USER MANAGER | Create / modify users, add users to roles | +| ROLE OWL ROLE MANAGER | Create roles, edit role mappings to users / AD groups / datasets | +| ROLE DATASET MANAGER | Create / modify datasets to roles, masking of dataset columns | +| ROLE OWL CHECK | Only role that can run DQ scans if Owlcheck security is enabled | +| ROLE DATA PREVIEW | Only role that can view source data if data preview security is enabled | +| ROLE DATASET TRAIN | Only role that can train datasets if dataset train security is enabled | +| ROLE DATASET RULES | Only role that can add / edit / delete rules if dataset rules security is enabled | +| ROLE PUBLIC | Public: Access to scorecards, no dataset access when dataset security is enabled | + +Custom roles can be added via the Role Management page by navigating to the Admin Console and clicking on the Roles Icon. Custom roles can also be added 'on the fly' during the Active Directory Role Mapping step. + +It is these custom roles that will determine the users that have access to datasets (including profile/rules/data preview/scoring), and database connections + +Additional information regarding setting up Dataset and Connection security can be found in those documents respectively. diff --git a/security/configuration/securing-passwords.md b/security/configuration/securing-passwords.md new file mode 100644 index 00000000..da5777d8 --- /dev/null +++ b/security/configuration/securing-passwords.md @@ -0,0 +1,22 @@ +# Securing Passwords + +Security is the utmost importance for Collibra DQ and our customers. In order to not send around plain text passwords when owlchecks are executed from the CLI users/admins can encrypt passwords and execute owlchecks using the encrypted passwords instead of plain text. To encrypt your password execute the command below\ + + +| owlmanage.sh encypt=password | +| ---------------------------- | + +The output password will look something like the following\ + + +| Q+Ri1S+ljpG+fDefXLY4/vXtUosspAoL | +| -------------------------------- | + +Now you can use this password in any owlcheck from the command line where you would normally place a plain text password. Example owlcheck with encrypted password in place of plain text password highlighted in bold.\ + + +| ./owlcheck -q "SELECT id, browser->'$.name' browser FROM events" -c "jdbc:mysql://54.212.36.218:2212/test" -u "owl" -p "Q+Ri1S+ljpG+fDefXLY4/vXtUosspAoL" -driver "com.mysql.cj.jdbc.Driver" -lib "/opt/owl/drivers/mysql8" -ds jsonremotemysql -rd "2019-01-28" | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + +\ +If executing a DQ Job from within the Owl-web the owl-web will automatically encrypt your password for you -- so you do not have to encrypt it. All passwords are masked out of the logs in order not to store plan passwords for security purposes. diff --git a/security/configuration/ssl-setup-https.md b/security/configuration/ssl-setup-https.md new file mode 100644 index 00000000..668fdb6d --- /dev/null +++ b/security/configuration/ssl-setup-https.md @@ -0,0 +1,36 @@ +# SSL Setup (HTTPS) + +By Default Collibra DQ has plain HTTP enabled for testing. When you are ready to enable SSL for the web application you can set the following environment variables in owl-env.sh to enable HTTPS. + +The settings listed at the bottom of this page will disable un-secure HTTP, enable secure HTTPS, and allow you to point to your certificate key store + credentials. \*A restart of the web-application is required. + +Before starting please have an accessible key store. + +``` +export SERVER_SSL_KEY_STORE: +``` + +You can call Collibra DQ's built in 256-bit encryption for the SERVER\_SSL\_KEY\_PASS value from the bin directory: ./owlmanage.sh encrypt=\ + +Use the response value instead of the plain text value to secure your password. + +``` +export SERVER_SSL_KEY_PASS: +``` + +``` +export SERVER_HTTP_ENABLED:false +export SERVER_HTTPS_ENABLED:true +export SERVER_REQUIRE_SSL:true + +####START KEYSTORE SETTINGS#### +export SERVER_SSL_KEY_TYPE:PKCS12 +#SET PATH TO KEYSTORE +export SERVER_SSL_KEY_STORE:KeystorePathHere +export SERVER_SSL_KEY_PASS:******* +export SERVER_SSL_KEY_ALIAS:keystoreAliasNameHere +``` + +Most common SSL types are JKS and PKCS12 + +\*Dont forget to restart the web application from the bin directory: ./owlmanage.sh restart=owlweb diff --git a/security/owl-security.md b/security/owl-security.md new file mode 100644 index 00000000..60fb493f --- /dev/null +++ b/security/owl-security.md @@ -0,0 +1,9 @@ +# Overview + +Owl offers multiple methods of user authentication including a local user store and Active Directory or generic LDAP integration. + +Security can be configured to meet your needs. Advanced options to segment groups and roles are available. Additionally, options for SAML and SSO are available. + +You can control configurations at the Web (UI), Postgres, and Application layers depending on your security requirements. Encryption is available for data in-transit or at-rest. + +![](<../.gitbook/assets/image (119).png>) diff --git a/workflows/assignment-queue-s/README.md b/workflows/assignment-queue-s/README.md new file mode 100644 index 00000000..1bc31ec6 --- /dev/null +++ b/workflows/assignment-queue-s/README.md @@ -0,0 +1,19 @@ +# Assignment Queue(s) + +![](../../.gitbook/assets/assignments.gif) + +OwlDQ provides observations that usually need review to validate. In many cases, it makes sense to assign the validation to a person that has access to the source data. Assignments can be handled by Owl internally, or an existing ServiceNow or JIRA queue when configured. + +When reviewing any observation you will notice a column called "Action + +![](<../../.gitbook/assets/image (34).png>) + +From the action dropdown, if you choose to validate a finding, you will also have the ability to assign that item to another Owl user for further investigation + +![](<../../.gitbook/assets/image (33).png>) + +If you do not choose an assignee (X), it will mark the item as valid, but unassigned: Acknowledged + +Your External Queues are where the source of the assignment should be tracked, OWDQ/JIRA/Serive Now are all options. OWLDQ is automatically configured, but you can configure others via the Admin Console and clicking on Assignment Queues. + +![](<../../.gitbook/assets/image (32).png>) diff --git a/workflows/assignment-queue-s/external-assignment.md b/workflows/assignment-queue-s/external-assignment.md new file mode 100644 index 00000000..8f23760e --- /dev/null +++ b/workflows/assignment-queue-s/external-assignment.md @@ -0,0 +1,14 @@ +# External assignment + +Collibra DQ has the ability to link to an Assignment Queue. You can assign any item to a user that has previously logged into the application and has a matching Service Now account. Choose the configured queue from the assignment drop-down list after selecting the **Validate** option from the actions drop-down list as previously described. + +![](<../../.gitbook/assets/Screen Shot 2020-07-07 at 5.12.22 AM.png>) + +{% hint style="warning" %} +To configure a queue, you must have ROLE\_ADMIN or ROLE\_CONNECTION\_MANAGER. +{% endhint %} + +1. Go to the **Admin Console** and click **Assignment Queues**.\ + ![](<../../.gitbook/assets/Screen Shot 2020-07-07 at 5.06.59 AM.png>) +2. **Add** or **Edit** a Service Now configuration from the corresponding page.\ + ![](<../../.gitbook/assets/Screen Shot 2020-07-07 at 5.16.11 AM.png>) diff --git a/workflows/assignment-queue-s/faq.md b/workflows/assignment-queue-s/faq.md new file mode 100644 index 00000000..1c4b89fa --- /dev/null +++ b/workflows/assignment-queue-s/faq.md @@ -0,0 +1,9 @@ +# FAQ + +## Can anyone assign an item? + +Anyone with access to the dataset and TRAIN role (if enabled) can assign an item. Users who have been assigned an item can resolve without the TRAIN role. + +## How do I add people to the assignment list + +Once a person logs into the Owl Application they will get a profile and become eligible for assignment. diff --git a/workflows/assignment-queue-s/internal-assignment.md b/workflows/assignment-queue-s/internal-assignment.md new file mode 100644 index 00000000..d7c1810b --- /dev/null +++ b/workflows/assignment-queue-s/internal-assignment.md @@ -0,0 +1,5 @@ +# Internal Assignment + +Owl has a built in Assignment Queue. You can assign any item to a user that has previously logged into the application. Simply choose OWLDQ from the assignment dropdown after choosing the "validate" option from the actions dropdown as previously described. + +![](<../../.gitbook/assets/Screen Shot 2020-07-07 at 5.08.15 AM.png>) diff --git a/workflows/email-alerts.md b/workflows/email-alerts.md new file mode 100644 index 00000000..99179e57 --- /dev/null +++ b/workflows/email-alerts.md @@ -0,0 +1,46 @@ +# Email Alerts + +![](../.gitbook/assets/alerts.gif) + +### Setting up an Email Server using the WebApp + +Navigate to OwlWeb URL. Then click the gear icon on the left in the admin section and click alerts. + +![](<../.gitbook/assets/Screen Shot 2020-07-27 at 9.09.04 PM.png>) + +### Creating a Condition to Send an Alert + +Also shows some of the common rules associated with this dataset for quick reference while creating. Use the batch name to create a consolidated list of alerts and distribution list for a set of notifications per dataset. + +![](<../.gitbook/assets/Screen Shot 2020-07-27 at 9.14.20 PM.png>) + +## DQ Alert in your Inbox + +Below is what an alert email that makes it into your inbox will look like. Always smart to make sure your email client didn't mark it as spam and that the SMTP was setup properly. Some email clients automatically strip images therefore we try to keep the emails light on styling and straight to the point. + +![](<../.gitbook/assets/Screen Shot 2021-08-11 at 9.46.48 PM.png>) + +## Failed DQ Job Email Alert + +There is also a scenario where the DQ job fails to run or has an exception and therefore never gets a chance to score the data or run the alert condition. This is is a FAILED alert. + +![](<../.gitbook/assets/Screen Shot 2021-08-25 at 10.07.43 AM.png>) + +## Alert Notifications in Web UI + +There are also alert notifications in the web UI. This can be helpful to confirm that the email alerts were sent out and to whom should have received the notifications. + +![](<../.gitbook/assets/Screen Shot 2021-08-12 at 8.03.07 AM.png>) + +### Setting up the EmailServer Programmatically + +If you are in a notebook or pipeline you may prefer to use the Scala/Spark API to create the Email Server + +```scala + val emailServer = OwlUtils.createEmailServer("smtp-relay.sendinblue.com", 587) + emailServer.setUsername("abc@owl-analytics.com") + emailServer.setPassword("abc") + emailServer.setCurrentSet(1) + Util.EmailServerDaoFactory.delete(emailServer) + Util.EmailServerDaoFactory.insert(emailServer) +```