Skip to content

fdac21/news

BranchesTags

Repository files navigation

Slow Server Please read

  • Everyone seems to be running MP3D now, so the server is barely keeping up. Two suggestions:
  • Please add nrow=10000 in your read.table (dat=read.table("features.gz", sep=";",comment.char="",quote="",nrow=10000)
  • if you completed MP3D, please shutdown your notebook to reduce RAM footprint (there is a large SHUTDOWN button next to each running notebook if yo select "running" tab)

TNVoice (TN Volunteer Online Instructor & Course Evaluations)

- https://utk.campuslabs.com
- Please let me know how you'd like to improve this course

Final Project Reports (Due Dec 9)

Similar to proposals, but note additional sections:

  • Objective (research question)
  • Data that was used: how obtained, how processed, integrated, and validated
  • What models or algorithms were used
  • Results: A description of the results
  • Primary issues encountered during the project
  • Future work: ideas generated, improvements that would make sense, etc
  • Org chart: rough timeline and responsibilities for each member

Presentations on Dec 9 (1:00 p.m. – 3:15 p.m.): all these presentations will be done remotely via zoom not in class

  • Superhero group
  • LeagueOfMythics
  • College Football Data Modeling
  • WhyJohnnyCantWinMedals
  • GPU-Market
  • SevenSeas
  • SteamScraper
  • LPC

Presentations on Nov 30

  • MovieBoxOfficePrediction
  • Fantasty football group
  • VideoGame_Analysis
  • YoutubeClickbait
  • ZillowAnalysis
  • LocationAnalysis

Presentations on Nov 23

  • Spotify
  • DIR
  • EAWOC

Presentations on Nov 18

  • FunnyNotFunny

Classes on Nov 16

  • Finish describing MP3D: data is ready: due Dec 9

Classes on Nov 11

  • Introduce MP3D
  • MISSING MP3A: dschult9=0,tainley=0
  • MISSING MP3B: abanks19 ahughe51 asuther8 dmaioran dschult9 fxie6 jhammer3 tainley
  • MISSING MP3C: All done: Thank you!

Classes on Nov 9

  • Work on final projects

Classes on Nov 4

  • !!!MP3C Now Operational!!!
  • Introduce MP3C Due Nov 11
  • Please comment on issue #9 regarding your preferred project presentation

Classes on Nov 2

  • Questions
  • Work on final projects
  • Missing Miniproject 2: rstewa35=1, spatel91=0
  • Missing Miniproject 3 part A: dschult9=0,hkitts2=0,mmohandi=0,tainley=0
  • Missing Miniproject 3 part B - has to be a) in your docker, b) in your home folder, c) files named yournetid.Object.stage.frame#.png: abanks19 ahughe51 asuther8 dmaioran dschult9 fxie6 hkitts2 jhammer3 jmatloc1 mmohandi rbray2 rstewa35 spatel91 tainley

Classes on Oct 26 and 28

  • MP3C=B due Oct 28
  • work on final project
  • I'll answer questions/provide help with gcp/MP3

Class on Oct 21

  • Introducing Miniproject 3 part B
  • MP3 Part A is not finished by (on Oct 20 at 23:11): abanks19=0,ahughe51=0,avo5=0,dlomax=0,dschult9=0,fxie6=0,hkitts2=0,jmatloc1=0,jmuncy2=0,mmohandi=0,mxw557=0,plin5=0,psherri1=597,rbray2=0,rstewa35=0,shwang6=0,spatel91=0,tainley=0,yli53=0
  • Questions
  • Work on final project

Class on Oct 19

  • no class due to engineering day: join if you are not done with your MP2/3, or have any questions

Class on Oct 14

  • Work on final projects
  • If there are any, I will answer questions on gcp, MP3
  • as of Oct 17, 20:55 status for MP3: abanks19=0,ahughe51=0,asuther8=764,avo5=0,dlomax=0,dschult9=0,fxie6=0,hkitts2=0,jmatloc1=0,jmoczadl=0,jmuncy2=0,jnotowic=0,mmohandi=0,mmuszik1=0,mxw557=0,plin5=0,ppatel91=0,psherri1=597,rbray2=0,rstewa35=0,shwang6=0,spatel91=0,tainley=0,tflorey=0,yli53=0

Class on Oct 12

  • Questions on MP2, gcp, MP3
  • Work on final projects

Class on Oct 7

  • Questions on gcp
  • Introducing MP3
  • Work on final projects
  • Still missing/too few for MP2 (on Oct 14, 15:49):ahughe51=0, jhammer3=0, jmatloc1=0, jmuncy2=0, rstewa35=0, spatel91=0

Class on Oct 5

  • Miniproject2 is due at the end of the class
  • Introducing GCP

Class on Sep 28

  • Final project proposals are due at the end of the class

  • The group needs to submit a project proposal (1.5-2 pages in IEEE format (see https://www.overleaf.com/latex/templates/preparation-of-papers-for-ieee-sponsored-conferences-and-symposia/zfnqfzzzxghk).

  • The proposal should provide

    an objective
a brief motivation for the project,
detailed discussion of the data that will be obtained or used in the project,
responsibilities of each member, along with
a time-line of milestones, and
the expected outcome

  • The proposal pdf will be committed to fdac21/ProjectName/proposal.pdf

  • Questions on proposal and MP2

Class on Sep 23

  • Lecture on data discovery and data storing (databases).
  • Introducing Miniproject2
  • Work on project proposal

Class on Sep 21 (in person class, please attend in person if you can)

  • Miniproject1 presentations selected on Sept 14: be ready to share it from your zoom session (if possible in-class) so that I can display on the board, record, share with students who could not come, and, most importantly, avoid the pain of hooking up cables.
  • The presentations will go in group order (the representative from the first group, the second group...)
  • Teams formed
    • I created fdac21/ProjectName repo and a team of the same name - please let me know if your project is missing or if you are not in project yet
    • I added the boaster creator as a member of the team: please invite the remaining members of the team)
  • Start brainstorming/writing final project proposal (see Sep 28 deadline)

Class on Sep 16 (zoom class)

  • Finish boasters (I see only four teams so far: need 12+more)/organize into teams
  • Submit final version of Miniproject1
  • Issues

Class "on Sep 14" feel free to set up a time with your sub-team before Sept 16 to

  • Practice presentations within groups: please either come in person or participate remotely. Please organize communications (if remote) within each group.
  • Please do not forget to select one presentation from each group to be presented for the entire class (https://github.com/fdac21/Miniproject1/issues/1)

Class on Sep 9

  • Class will be online-only (see zoom info below)

  • Make sure you have

    a. Forked fdac21/Miniproject1
b. Posted the idea for your analysis on your peer's fork
c. Responded to the idea that was posted by your peer

  • Boasters for class project

  • Question regarding MP1

Class on Sep 7

  • Class will be online-only (see zoom info below)
  • See the simple text analysis of your descriptions
  • Introducing the MiniProject1 process and template
  • Boasters for class project

Class on Sep 2

  • Class will be online-only (see zoom info) because parking lots need to be empty by 3PM and building will be locked!
  • Critical Tools
  • Version Control

Class on Aug 31: Class will be online-only (see zoom info below)

Class on Aug 26 !!Class is cancelled today!!

Class on Aug 24

  • Please accept your invitation to fdac21 organization while logged in to GH via handle you used to submit pull request
  • If you have not done so yet, please accept github fdac21 invitation (ahillhou)
  • then
    • fork repo students
    • create your utid.md file providing your name and interests: see per fdac21/students/README.md, and also provide your utid.key with your public ssh key. Once done, please
    • submit a pull request to fdac21/students
    • still missing.md files: ahillhou, jmuncy2
  • Introductory lecture

Class on Aug 19

  • Create your github account
    • fork repo students
    • create your utid.md file providing your name and interests: see per fdac21/students/README.md, and also provide your utid.key with your public ssh key. Once done, please
    • submit a pull request to fdac21/students
  • Make sure you do it a day before the next class so we can start ready

Information for remote participation via Zoom

Syllabus for "Fundamentals of Digital Archeology"

  • Course: [COSCS-445/COSCS-545]
  • ** MK404 and Zoom link above **
  • ** TTh 4:30-5:45
  • Instructor: Audris Mockus, [email protected] office hours - upon request
  • TA: James Hammer (jhammer3) and Dylan Lomax (dlomax). Office hours - upon request
  • Need help?

Simple rules:

  1. There are no stupid questions. However, it may be worth going over the following steps:
  2. Think of what the right answer may be.
  3. Search online: stack overflow, etc.
  4. Look through issues
  5. Post the question as an issue.
  6. Ask instructor: email for 1-on-1 help, or to set up a time to meet

Objectives

The course will combine theoretical underpinning of big data with intense practice. In particular, approaches to ethical concerns, reproducibility of the results, absence of context, missing data, and incorrect data will be both discussed and practiced by writing programs to discover the data in the cloud, to retrieve it by scraping the deep web, and by structuring, storing, and sampling it in a way suitable for subsequent decision making. At the end of the course students will be able to discover, collect, and clean digital traces, to use such traces to construct meaningful measures, and to create tools that help with decision making.

Expected Outcomes

Upon completion, students will be able to discover, gather, and analyze digital traces, will learn how to avoid mistakes common in the analysis of low-quality data, and will have produced a working analytics application.

In particular, in addition to practicing critical thinking, students will acquire the following skills:

  • Use Python and other tools to discover, retrieve, and process data.

  • Use data management techniques to store data locally and in the cloud.

  • Use data analysis methods to explore data and to make predictions.

Course Description

A great volume of complex data is generated as a result of human activities, including both work and play. To exploit that data for decision making it is necessary to create software that discovers, collects, and integrates the data.

Digital archeology relies on traces that are left over in the course of ordinary activities, for example the logs generated by sensors in mobile phones, the commits in version control systems, or the email sent and the documents edited by a knowledge worker. Understanding such traces is complicated in contrast to data collected using traditional measurement approaches.

Traditional approaches rely on a highly controlled and well-designed measurement system. In meteorology, for example, the temperature is taken in specially designed and carefully selected locations to avoid direct sunlight and to be at a fixed distance from the ground. Such measurement can then be trusted to represent these controlled conditions and the analysis of such data is, consequently, fairly straightforward.

The measurements from geolocation or other sensors in mobile phones are affected by numerous (yet not recorded) factors: was the phone kept in the pocket, was it indoors or outside? The devices are not calibrated or may not work properly, so the corresponding measurements would be inaccurate. Locations (without mobile phones) may not have any measurement, yet may be of the greatest interest. This lack of context and inaccurate or missing data necessitates fundamentally new approaches that rely on patterns of behavior to correct the data, to fill in missing observations, and to elucidate unrecorded context factors. These steps are needed to obtain meaningful results from a subsequent analysis.

The course will cover basic principles and effective practices to increase the integrity of the results obtained from voluminous but highly unreliable sources.

  • Ethics: legal aspects, privacy, confidentiality, governance

  • Reproducibility: version control, ipython notebook

  • Fundamentals of big data analysis: extreme distributions, transformations, quantiles, sampling strategies, and logistic regression

  • The nature of digital traces: lack of context, missing values, and incorrect data

Prerequisites

Students are expected to have basic programming skills, in particular, be able to use regular expressions, programming concepts such as variables, functions, loops, and data structures like lists and dictionaries (for example, COSC 365)

Being familiar with version control systems (e.g., COSC 340), Python (e.g., COSC 370), and introductory level probability (e.g., ECE 313) and statistics, such as, random variables, distributions and regression would be beneficial but is not expected. Everyone is expected, however, to be willing and highly motivated to catch up in the areas where they have gaps in the relevant skills.

All the assignments and projects for this class will use github and Python. Knowledge of Python is not a prerequisite for this course, provided you are comfortable learning on your own as needed. While we have strived to make the programming component of this course straightforward, we will not devote much time to teaching programming, Python syntax, or any of the libraries and APIs. You should feel comfortable with:

  1. How to look up Python syntax on Google and StackOverflow.
  2. Basic programming concepts like functions, loops, arrays, dictionaries, strings, and if statements.
  3. How to learn new libraries by reading documentation and reusing examples
  4. Asking questions on StackOverflow or as a GitHub issue.

Requirements

These apply to real life, as well.

  • Must apply "good programming style" learned in class
    • Optimize for readability
  • Bonus points for:
    • Creativity (as long as requirements are fulfilled)

Teaming Tips

  • Agree on an editor and environment that you're comfortable with
  • The person who's less experienced/comfortable should have more keyboard time
  • Switch who's "driving" regularly
  • Make sure to save the code and send it to others on the team

Evaluation

  • Class Participation – 15%: students are expected to read all material covered in a week and come to class prepared to take part in the classroom discussions (online). Asking and responding to other student questions (issues) counts as a key factor for classroom participation. With online format and collaborative nature of the projects, this should not be hard to accomplish.

  • Assignments - 40%: Each assignment will involve writing (or modifying a template of) a small Python program.

  • Project - 45%: one original project done alone or in a group of 2 or 3 students. The project will explore one or more of the themes covered in the course that students find particularly compelling. The group needs to submit a project proposal (2 pages IEEE format) approximately 1.5 months before the end of term. The proposal should provide a brief motivation of the project, detailed discussion of the data that will be obtained or used in the project, along with a time-line of milestones, and expected outcome.

Other considerations

As a programmer you will never write anything from scratch, but will reuse code, frameworks, or ideas. You are encouraged to learn from the work of your peers. However, if you don't try to do it yourself, you will not learn. deliberate-practice (activities designed for the sole purpose of effectively improving specific aspects of an individual's performance) is the only way to reach perfection.

Please respect the terms of use and/or license of any code you find, and if you re-implement or duplicate an algorithm or code from elsewhere, credit the original source with an inline comment.

Resources

Materials

This class assumes you are confident with this material, but in case you need a brush-up...

Other

Databases
  • A MongoDB Schema Analyzer. One JavaScript file that you run with the mongo shell command on a database collection and it attempts to come up with a generalized schema of the datastore. It was also written about on the official MongoDB blog.
R and data analysis
  • Modern Applied Statistics with S (4th Edition) by William N. Venables, Brian D. Ripley. ISBN0387954570
  • R
  • Code School
  • Quick-R
Tutorials written as ipython-notebooks

GitHub

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

6 stars

Watchers

5 watching

Forks

10 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •