exploring_HN_posts.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Exploring Hacker News Posts.ipynb","provenance":[],"collapsed_sections":[],"mount_file_id":"1uOsrPZUnT0wAhkNLqRBAMSsYgekVpOKP","authorship_tag":"ABX9TyO7CmZVx4nBYOv6gmqnc6EM"},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"6PztQOjS30Zq","colab_type":"text"},"source":["# Exploring Hacker News Posts"]},{"cell_type":"markdown","metadata":{"id":"CTMhd5JS36dw","colab_type":"text"},"source":["## Aim\n","\n","The aim of this project is to investigate the features of a post that predict the number of comments/votes it will attract and gain insight into the type and timing of posting that attracts most comments on Hacker News. We will look at which type of posts `Ask HN` or `Show HN` receive on average the highest number of comments and then analyze the type with the higher average in more detail by determining the average number of comments for posts at each hour of the day (adjusted for my current timezone in Romania!) We will then do a similar analysis of the number of points per post. "]},{"cell_type":"markdown","metadata":{"id":"TTadYc3br9YV","colab_type":"text"},"source":["\n","## About Hacker News\n","\n","[Hacker News](https://news.ycombinator.com/) (HN) is a website started by the startup incubator [Y Combinator](https://www.ycombinator.com/), where user-submitted stories (known as \"posts\") are voted and commented upon, similar to reddit. Hacker News is extremely popular in technology and startup circles, and posts that make it to the top of Hacker News' listings can get hundreds of thousands of visitors as a result.\n","\n"]},{"cell_type":"markdown","metadata":{"id":"h5eh2Eww3Trn","colab_type":"text"},"source":["## Full dataset\n","\n","The [full dataset](https://www.kaggle.com/hacker-news/hacker-news-posts) of approx. 300,000 rows from which the reduced dataset of approx. 20,000 rows used for this project was extracted is Hacker News posts from the 12 months up to September 26 2016. The original dataset was reduced by removing all submissions that did not receive any comments, and then randomly sampling from the remaining submissions. \n","\n","The dataset includes the following columns:\n","\n","* title: title of the post (self explanatory)\n","\n","* url: the url of the item being linked to\n","\n","* num_points: the number of upvotes the post received\n","\n","* num_comments: the number of comments the post received\n","\n","* author: the name of the account that made the post\n","\n","* created_at: the date and time the post was made (the time zone is Eastern Time in the US)\n","\n","[GitHub link](https://github.com/minimaxir/get-all-hacker-news-submissions-comments) to HN scraper. "]},{"cell_type":"markdown","metadata":{"id":"F3F5OPEe5bSD","colab_type":"text"},"source":["## Reading the reduced dataset csv file\n","\n","The reduced dataset used in this project is stored in the file 'hacker_news.csv'. We open this file and read it into the workspace, assigning as a list of lists to the variable `hn`. "]},{"cell_type":"code","metadata":{"id":"MyBnl7Am6CaD","colab_type":"code","outputId":"874da8f3-780f-45ae-d73b-41c883e76e3b","executionInfo":{"status":"ok","timestamp":1587636023961,"user_tz":-180,"elapsed":3395,"user":{"displayName":"Thomas Haysom","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhBs8NDv6W64WAgLqNEKHcbH9D_ao3hTsLjzHnmNw=s64","userId":"07768989857857358003"}},"colab":{"base_uri":"https://localhost:8080/","height":93}},"source":["import csv\n","\n","opened_file = open('/content/drive/My Drive/Datasets/hacker_news.csv')\n","read_file = csv.reader(opened_file)\n","hn = list(read_file)\n","hn_headers = hn[0]\n","hn = hn[1:]\n","\n","# display the headers row\n","print(hn_headers)\n","print()\n","# display first 5 posts in database\n","print(hn[:5])"],"execution_count":1,"outputs":[{"output_type":"stream","text":["['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at']\n","\n","[['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', \"Florida DJs May Face Felony for April Fools' Water Joke\", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01'], ['10301696', 'Note by Note: The Making of Steinway L1037 (2007)', 'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0', '8', '2', 'walterbell', '9/30/2015 4:12']]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"EB1Qan2_7xmx","colab_type":"text"},"source":["## Separate types of post into 'Ask HN', 'Show HN' and other\n","\n","Some post titles begin with either `Ask HN` or `Show HN`. Users submit `Show HN` posts to show the HN community a project, product, or just generally something interesting. Likewise, users submit `Ask HN` posts to ask the HN community a specific question. We will investigate which of these post types receives more comments on average. \n","\n","We begin by separating the dataset into 3 lists called `ask_posts`, `show_posts` and `other_posts`:\n","\n"]},{"cell_type":"code","metadata":{"id":"P4wRzSNI8-Me","colab_type":"code","outputId":"e6bcb614-71e9-45e1-abb6-18cf071e3ee7","executionInfo":{"status":"ok","timestamp":1587636023963,"user_tz":-180,"elapsed":3347,"user":{"displayName":"Thomas Haysom","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhBs8NDv6W64WAgLqNEKHcbH9D_ao3hTsLjzHnmNw=s64","userId":"07768989857857358003"}},"colab":{"base_uri":"https://localhost:8080/","height":73}},"source":["ask_posts, show_posts, other_posts = [], [], []\n","\n","for row in hn:\n","  title = row[1]\n","  if title.lower().startswith('ask hn'):\n","    ask_posts.append(row)\n","  elif title.lower().startswith('show hn'):\n","    show_posts.append(row)\n","  else:\n","    other_posts.append(row)\n","\n","# display number of posts in each list\n","print('Number of posts with title beginning \\'Ask HN\\': ',len(ask_posts))\n","print('Number of posts with title beginning \\'Show HN\\': ',len(show_posts))\n","print('Number of posts with title beginning with neither : ',len(other_posts))"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Number of posts with title beginning 'Ask HN':  1744\n","Number of posts with title beginning 'Show HN':  1162\n","Number of posts with title beginning with neither :  17194\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"H7_LmGKVGdIj","colab_type":"text"},"source":["We will compare the average number of comments for `Ask HN` posts vs. `Show HN` posts by creating a function to return the average number of comments for posts in a list given as argument:"]},{"cell_type":"code","metadata":{"id":"NWkNdOAPHNiK","colab_type":"code","outputId":"dae71fbe-2140-4210-825e-90417f5d9534","executionInfo":{"status":"ok","timestamp":1587636023964,"user_tz":-180,"elapsed":3336,"user":{"displayName":"Thomas Haysom","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhBs8NDv6W64WAgLqNEKHcbH9D_ao3hTsLjzHnmNw=s64","userId":"07768989857857358003"}},"colab":{"base_uri":"https://localhost:8080/","height":73}},"source":["def average_comments(posts):\n","  total_comments = 0\n","  for post in posts:\n","    total_comments += int(post[4])\n","  \n","  num_posts = len(posts)\n","  return total_comments / num_posts\n","\n","avg_ask_comments = average_comments(ask_posts)\n","avg_show_comments = average_comments(show_posts)\n","avg_other_comments = average_comments(other_posts)\n","\n","# display average comments for types of post\n","print('Average number of comments for posts with title beginning \\'Ask HN\\': ', avg_ask_comments)\n","print('Average number of comments for posts with title beginning \\'Show HN\\': ', avg_show_comments)\n","print('Average number of comments for other posts: ', avg_other_comments)"],"execution_count":3,"outputs":[{"output_type":"stream","text":["Average number of comments for posts with title beginning 'Ask HN':  14.038417431192661\n","Average number of comments for posts with title beginning 'Show HN':  10.31669535283993\n","Average number of comments for other posts:  26.8730371059672\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"6Uq1KK1DJO-T","colab_type":"text"},"source":["The mean number of comments on posts beginning `Ask_HN` is greater than the mean number of comments on posts beginning `Show_HN`. Since `Ask_HN` posts are more likely to receive comments we will focus on these posts in more detail."]},{"cell_type":"markdown","metadata":{"id":"vz-nKhk0KRFl","colab_type":"text"},"source":["## How does the time a post is created affect the number of comments?\n","\n","We will determine if ask posts created at a certain time are more likely to attract comments. For each hour of the day, we will calculate the number of posts and the total number of comments for these posts and use these values to calculate the average number of comments for posts created by each hour of the day. \n","\n","By displaying the value of the `created_at` column for the first 10 posts in the `hn` list of lists, with reference to a Python strftime [format guide](https://strftime.org/), we can deduce the format of the dates and times in the `created_at` column is `'%-m/%-d/%Y %-H:%M'`  "]},{"cell_type":"code","metadata":{"id":"bvcpIPssLxNe","colab_type":"code","outputId":"9e637810-07f8-44a6-9a5a-2f3f462a9fca","executionInfo":{"status":"ok","timestamp":1587636023966,"user_tz":-180,"elapsed":3311,"user":{"displayName":"Thomas Haysom","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhBs8NDv6W64WAgLqNEKHcbH9D_ao3hTsLjzHnmNw=s64","userId":"07768989857857358003"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["for post in hn[:10]:\n","  print(post[-1])"],"execution_count":4,"outputs":[{"output_type":"stream","text":["8/4/2016 11:52\n","1/26/2016 19:30\n","6/23/2016 22:20\n","6/17/2016 0:01\n","9/30/2015 4:12\n","10/31/2015 9:48\n","11/13/2015 0:45\n","8/16/2016 9:55\n","3/22/2016 16:18\n","10/13/2015 9:30\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"CjNs-c4zK8A6","colab_type":"code","outputId":"4e43e3f2-8eae-472b-f701-6c91b3ac49ea","executionInfo":{"status":"ok","timestamp":1587636023967,"user_tz":-180,"elapsed":3297,"user":{"displayName":"Thomas Haysom","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhBs8NDv6W64WAgLqNEKHcbH9D_ao3hTsLjzHnmNw=s64","userId":"07768989857857358003"}},"colab":{"base_uri":"https://localhost:8080/","height":186}},"source":["import datetime as dt\n","\n","strftime_format = '%m/%d/%Y %H:%M'\n","result_list = []\n","\n","for row in ask_posts: # iterate over ask_posts\n","  created_at_string = row[-1]\n","  created_at_dt = dt.datetime.strptime(created_at_string, strftime_format)\n","  num_comments = int(row[4])\n","  result_list.append([created_at_dt, num_comments])\n","\n","counts_by_hour = {}\n","comments_by_hour = {}\n","\n","for result in result_list:\n","  if str(result[0].hour) in counts_by_hour:\n","    counts_by_hour[str(result[0].hour)] += 1\n","    comments_by_hour[str(result[0].hour)] += result[1]\n","  else:\n","    counts_by_hour[str(result[0].hour)] = 1\n","    comments_by_hour[str(result[0].hour)] = result[1]\n","\n","# initialize and populate a list of lists where each element is a list containing\\\n","# the hour of day and the average number of comments for posts made during that hour\n","avg_by_hour = []\n","for hour in counts_by_hour:\n","  avg = comments_by_hour[hour] / counts_by_hour[hour]\n","  avg_by_hour.append([hour, avg])\n","\n","# order the list by desc average number of comments and display\n","sorted_avg_by_hour = sorted(avg_by_hour, key=lambda x: x[1], reverse=True)\n","print(sorted_avg_by_hour)\n","\n","# display top 5 hours with highest average number of comments for 'Ask HN' posts\n","print()\n","print('Top 5 hours for \\'Ask HN\\' post comments')\n","template = '{hour}: average number of comments per post: {average:.2f}'\n","for item in sorted_avg_by_hour[:5]:\n","  hour_dt = dt.datetime.strptime(item[0], '%H')\n","  print(template.format(hour=hour_dt.strftime('%H:%M'), average = item[1]))  \n"],"execution_count":5,"outputs":[{"output_type":"stream","text":["[['15', 38.5948275862069], ['2', 23.810344827586206], ['20', 21.525], ['16', 16.796296296296298], ['21', 16.009174311926607], ['13', 14.741176470588234], ['10', 13.440677966101696], ['14', 13.233644859813085], ['18', 13.20183486238532], ['17', 11.46], ['1', 11.383333333333333], ['11', 11.051724137931034], ['19', 10.8], ['8', 10.25], ['5', 10.08695652173913], ['12', 9.41095890410959], ['6', 9.022727272727273], ['0', 8.127272727272727], ['23', 7.985294117647059], ['7', 7.852941176470588], ['3', 7.796296296296297], ['4', 7.170212765957447], ['22', 6.746478873239437], ['9', 5.5777777777777775]]\n","\n","Top 5 hours for 'Ask HN' post comments\n","15:00: average number of comments per post: 38.59\n","02:00: average number of comments per post: 23.81\n","20:00: average number of comments per post: 21.52\n","16:00: average number of comments per post: 16.80\n","21:00: average number of comments per post: 16.01\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"bWQnAAVMw2MR","colab_type":"text"},"source":["We can see that posts published to HN between 15:00 and 16:00 hours (Eastern Time) receive the highest mean number of comments. The timezone of the time values in the `created_at` column of the dataset is US Eastern Time (ET) (UTC -5h). I am currently in Romania (land of Dracula and lots and lots of meat!) so my timezone is Eastern European Standard Time (EEST) (UTC +2h) so we will print out the top 5 hours for average number of post comments adjusting for this time difference:"]},{"cell_type":"code","metadata":{"id":"MdzJHCgeyF-6","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":148},"outputId":"e5ca6386-fc16-4437-8679-23dd239d60f2","executionInfo":{"status":"ok","timestamp":1587636023968,"user_tz":-180,"elapsed":3282,"user":{"displayName":"Thomas Haysom","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhBs8NDv6W64WAgLqNEKHcbH9D_ao3hTsLjzHnmNw=s64","userId":"07768989857857358003"}}},"source":["print()\n","print('Top 5 hours for \\'Ask HN\\' post comments in EEST')\n","template = '{hour}: average number of comments per post: {average:.2f}'\n","for item in sorted_avg_by_hour[:5]:\n","  new_hour = (int(item[0]) + 7) % 24 # modular 24 value required\n","  hour_dt = dt.datetime.strptime(str(new_hour), '%H')\n","  print(template.format(hour=hour_dt.strftime('%H:%M'), average = item[1])) "],"execution_count":6,"outputs":[{"output_type":"stream","text":["\n","Top 5 hours for 'Ask HN' post comments in EEST\n","22:00: average number of comments per post: 38.59\n","09:00: average number of comments per post: 23.81\n","03:00: average number of comments per post: 21.52\n","23:00: average number of comments per post: 16.80\n","04:00: average number of comments per post: 16.01\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"3--nMlMBhBrO","colab_type":"text"},"source":["We can see that the data is telling us that the best hour of day to post a `Ask HN` post to maximize the number of comments received is between 10pm and 11pm in Romania. `Ask HN` posts published during this hour receive on average over 50% more comments than `Ask HN` posts published during any other single hour of the day! Recall that we have only analyzed the `Ask HN` posts by hour of day (since these posts received more comments on average than `Show HN` posts) so the conclusion could be different for non `Ask HN` posts. "]},{"cell_type":"markdown","metadata":{"id":"c34LY5nFc106","colab_type":"text"},"source":["## Analyzing the number of points different posts receive\n","\n","We have analyzed at the number of comments received by different types of posts and the average number of posts for each hour of the day for `Ask HN` posts. We will now repeat a similar analysis for the number of points received by HN posts:"]},{"cell_type":"code","metadata":{"id":"AVmu452udmTf","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":73},"outputId":"03ca4d4f-2b1d-4fc4-b80e-84ca618c5ffb","executionInfo":{"status":"ok","timestamp":1587636023969,"user_tz":-180,"elapsed":3276,"user":{"displayName":"Thomas Haysom","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhBs8NDv6W64WAgLqNEKHcbH9D_ao3hTsLjzHnmNw=s64","userId":"07768989857857358003"}}},"source":["def average_points(posts):\n","  total_points = 0\n","  for post in posts:\n","    total_points += int(post[3])\n","  \n","  num_posts = len(posts)\n","  return total_points / num_posts\n","\n","avg_ask_points = average_points(ask_posts)\n","avg_show_points = average_points(show_posts)\n","avg_other_points = average_points(other_posts)\n","\n","# display average points for types of posts\n","print('Average number of points for posts with title beginning \\'Ask HN\\': ', avg_ask_points)\n","print('Average number of points for posts with title beginning \\'Show HN\\': ', avg_show_points)\n","print('Average number of points for other posts: ', avg_other_points)"],"execution_count":7,"outputs":[{"output_type":"stream","text":["Average number of points for posts with title beginning 'Ask HN':  15.061926605504587\n","Average number of points for posts with title beginning 'Show HN':  27.555077452667813\n","Average number of points for other posts:  55.4067698034198\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"0yk4QqUcfLfY","colab_type":"text"},"source":["It appears that `Show HN` posts receive more points on average than `Ask HN` posts, so we will calculate an hourly average number of points for this subset of the dataset. \n","\n"]},{"cell_type":"code","metadata":{"id":"iqjcjdK5flgz","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":186},"outputId":"4454a2b5-e333-4fe4-eeab-2caedc8b4863","executionInfo":{"status":"ok","timestamp":1587636389446,"user_tz":-180,"elapsed":1545,"user":{"displayName":"Thomas Haysom","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhBs8NDv6W64WAgLqNEKHcbH9D_ao3hTsLjzHnmNw=s64","userId":"07768989857857358003"}}},"source":["# datetime module already imported\n","\n","# strftime_format = '%m/%d/%Y %H:%M'\n","result_list = []\n","\n","for row in show_posts: # iterate over show_posts\n","  created_at_string = row[-1]\n","  created_at_dt = dt.datetime.strptime(created_at_string, strftime_format)\n","  num_points = int(row[3])\n","  result_list.append([created_at_dt, num_points])\n","\n","counts_by_hour = {}\n","points_by_hour = {}\n","\n","for result in result_list:\n","  if str(result[0].hour) in counts_by_hour:\n","    counts_by_hour[str(result[0].hour)] += 1\n","    points_by_hour[str(result[0].hour)] += result[1]\n","  else:\n","    counts_by_hour[str(result[0].hour)] = 1\n","    points_by_hour[str(result[0].hour)] = result[1]\n","\n","# initialize and populate a list of lists where each element is a list containing\\\n","# the hour of day and the average number of points for posts made during that hour\n","avg_by_hour = []\n","for hour in counts_by_hour:\n","  avg = points_by_hour[hour] / counts_by_hour[hour]\n","  avg_by_hour.append([hour, avg])\n","\n","# order the list by desc average number of points and display\n","sorted_avg_by_hour = sorted(avg_by_hour, key=lambda x: x[1], reverse=True)\n","print(sorted_avg_by_hour)\n","\n","# display top 5 hours with highest average number of points for 'Show HN' posts\n","print()\n","print('Top 5 hours for \\'Show HN\\' post points')\n","template = '{hour}: average number of points per post: {average:.2f}'\n","for item in sorted_avg_by_hour[:5]:\n","  hour_dt = dt.datetime.strptime(item[0], '%H')\n","  print(template.format(hour=hour_dt.strftime('%H:%M'), average = item[1]))  \n"],"execution_count":8,"outputs":[{"output_type":"stream","text":["[['23', 42.388888888888886], ['12', 41.68852459016394], ['22', 40.34782608695652], ['0', 37.83870967741935], ['18', 36.31147540983606], ['11', 33.63636363636363], ['19', 30.945454545454545], ['20', 30.316666666666666], ['15', 28.564102564102566], ['16', 28.322580645161292], ['17', 27.107526881720432], ['14', 25.430232558139537], ['3', 25.14814814814815], ['1', 25.0], ['13', 24.626262626262626], ['6', 23.4375], ['7', 19.0], ['10', 18.916666666666668], ['9', 18.433333333333334], ['21', 18.425531914893618], ['8', 15.264705882352942], ['4', 14.846153846153847], ['2', 11.333333333333334], ['5', 5.473684210526316]]\n","\n","Top 5 hours for 'Ask HN' post points\n","23:00: average number of points per post: 42.39\n","12:00: average number of points per post: 41.69\n","22:00: average number of points per post: 40.35\n","00:00: average number of points per post: 37.84\n","18:00: average number of points per post: 36.31\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"5Kn5JK2pgTpq","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":148},"outputId":"d954968b-48df-4dc8-f274-1b8b66a73fa2","executionInfo":{"status":"ok","timestamp":1587636514651,"user_tz":-180,"elapsed":1547,"user":{"displayName":"Thomas Haysom","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhBs8NDv6W64WAgLqNEKHcbH9D_ao3hTsLjzHnmNw=s64","userId":"07768989857857358003"}}},"source":["# adjusting to the land of Dracula (I waaaanntt to succckkk youur blooood...) timezone\n","print()\n","print('Top 5 hours for \\'Show HN\\' post points in EEST')\n","template = '{hour}: average number of points per post: {average:.2f}'\n","for item in sorted_avg_by_hour[:5]:\n","  new_hour = (int(item[0]) + 7) % 24 # modular 24 value required\n","  hour_dt = dt.datetime.strptime(str(new_hour), '%H')\n","  print(template.format(hour=hour_dt.strftime('%H:%M'), average = item[1])) "],"execution_count":10,"outputs":[{"output_type":"stream","text":["\n","Top 5 hours for 'Ask HN' post points in EEST\n","06:00: average number of points per post: 42.39\n","19:00: average number of points per post: 41.69\n","05:00: average number of points per post: 40.35\n","07:00: average number of points per post: 37.84\n","01:00: average number of points per post: 36.31\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"nyygWjQohiAt","colab_type":"text"},"source":["`Show HN` posts published between 6am and 7am EEST receive on average the highest number of points but there is not much difference between the average number of points per post for the top 5 hours in the list (the number 1 hour for posting has approx 17% more points per post than the number 5 hour)."]},{"cell_type":"markdown","metadata":{"id":"51zCHDBejLh7","colab_type":"text"},"source":["## Analyzing 'other' type posts\n","\n","In this analysis `other_posts` is a subset of rows from the dataset which contains posts with a title which does not begin with 'Ask HN' or 'Show HN', we refer to these as 'other' posts.\n","\n","We redisplay earlier analysis below:"]},{"cell_type":"code","metadata":{"id":"PxChChp3jLtg","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":166},"outputId":"edb82ea1-b748-473d-d1e4-c8b20723811b","executionInfo":{"status":"ok","timestamp":1587637225887,"user_tz":-180,"elapsed":2205,"user":{"displayName":"Thomas Haysom","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhBs8NDv6W64WAgLqNEKHcbH9D_ao3hTsLjzHnmNw=s64","userId":"07768989857857358003"}}},"source":["print('Comments: ')\n","# display average comments for types of post\n","print('Average number of comments for posts with title beginning \\'Ask HN\\': ', avg_ask_comments)\n","print('Average number of comments for posts with title beginning \\'Show HN\\': ', avg_show_comments)\n","print('Average number of comments for other posts: ', avg_other_comments)\n","print()\n","print('Points: ')\n","# display average points for types of posts\n","print('Average number of points for posts with title beginning \\'Ask HN\\': ', avg_ask_points)\n","print('Average number of points for posts with title beginning \\'Show HN\\': ', avg_show_points)\n","print('Average number of points for other posts: ', avg_other_points)"],"execution_count":11,"outputs":[{"output_type":"stream","text":["Comments: \n","Average number of comments for posts with title beginning 'Ask HN':  14.038417431192661\n","Average number of comments for posts with title beginning 'Show HN':  10.31669535283993\n","Average number of comments for other posts:  26.8730371059672\n","Points: \n","Average number of points for posts with title beginning 'Ask HN':  15.061926605504587\n","Average number of points for posts with title beginning 'Show HN':  27.555077452667813\n","Average number of points for other posts:  55.4067698034198\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"mYErA_vokO6e","colab_type":"text"},"source":["It is clear that other posts receive significantly more comments per post and more points per post than `Ask HN` and `Show HN` type posts. \n","\n","We will repeat the earlier processes of breaking down by hour the comments per post and the points per post for posts in the `other_posts` subset of the dataset: "]},{"cell_type":"code","metadata":{"id":"RRIF7oCooM6o","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":502},"outputId":"7289ba4f-0600-4886-8290-5029d5d9e4f8","executionInfo":{"status":"ok","timestamp":1587639409368,"user_tz":-180,"elapsed":1187,"user":{"displayName":"Thomas Haysom","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhBs8NDv6W64WAgLqNEKHcbH9D_ao3hTsLjzHnmNw=s64","userId":"07768989857857358003"}}},"source":["# datetime module already imported\n","\n","# strftime_format = '%m/%d/%Y %H:%M'\n","result_list = []\n","\n","for row in other_posts: # iterate over other_posts\n","  created_at_string = row[-1]\n","  created_at_dt = dt.datetime.strptime(created_at_string, strftime_format)\n","  num_comments = int(row[4])\n","  num_points = int(row[3])\n","  result_list.append([created_at_dt, num_comments, num_points])\n","\n","counts_by_hour = {}\n","points_by_hour = {}\n","comments_by_hour = {}\n","\n","for result in result_list:\n","  if str(result[0].hour) in counts_by_hour:\n","    counts_by_hour[str(result[0].hour)] += 1\n","    comments_by_hour[str(result[0].hour)] += result[1]\n","    points_by_hour[str(result[0].hour)] += result[2]\n","  else:\n","    counts_by_hour[str(result[0].hour)] = 1\n","    comments_by_hour[str(result[0].hour)] = result[1]\n","    points_by_hour[str(result[0].hour)] = result[2]\n","\n","# initialize and populate a list of lists where each element is a list containing\\\n","# the hour of day, the average number of points and the average number of comments\\\n","# for posts made during that hour\n","avg_by_hour = []\n","for hour in counts_by_hour:\n","  avg_comments = comments_by_hour[hour] / counts_by_hour[hour]\n","  avg_points = points_by_hour[hour] / counts_by_hour[hour]\n","  avg_by_hour.append([hour, avg_comments, avg_points])\n","\n","# order the list by desc average of number of points per post and number of\\\n","# comments per post\n","sorted_avg_by_hour = sorted(avg_by_hour, key=lambda x: (x[1] + x[2]) / 2, reverse=True)\n","\n","# display by hour (adjusted to EEST from ET) average of number of points per post and number of\\\n","# comments per post ordered desc\n","print()\n","print('Average of comments per post and points per post for \\'other\\' posts by hour (EEST) ordered desc: ')\n","template = '{hour}: average of comments and points per post: {average:.2f}'\n","for item in sorted_avg_by_hour[:24]:\n","  new_hour = (int(item[0]) + 7) % 24 # modular 24 value required\n","  hour_dt = dt.datetime.strptime(str(new_hour), '%H')\n","  avg_comments_points = (item[1] + item[2]) / 2\n","  print(template.format(hour=hour_dt.strftime('%H:%M'), average = avg_comments_points))  \n"],"execution_count":17,"outputs":[{"output_type":"stream","text":["\n","Average of comments per post and points per post for 'other' posts by hour (EEST) ordered desc: \n","21:00: average of comments and points per post: 47.06\n","20:00: average of comments and points per post: 46.71\n","22:00: average of comments and points per post: 45.03\n","19:00: average of comments and points per post: 43.87\n","18:00: average of comments and points per post: 43.58\n","17:00: average of comments and points per post: 43.55\n","02:00: average of comments and points per post: 43.36\n","09:00: average of comments and points per post: 43.13\n","00:00: average of comments and points per post: 42.99\n","07:00: average of comments and points per post: 42.77\n","10:00: average of comments and points per post: 41.87\n","14:00: average of comments and points per post: 41.82\n","16:00: average of comments and points per post: 40.76\n","15:00: average of comments and points per post: 40.56\n","01:00: average of comments and points per post: 40.43\n","23:00: average of comments and points per post: 39.79\n","06:00: average of comments and points per post: 38.32\n","12:00: average of comments and points per post: 37.57\n","11:00: average of comments and points per post: 36.90\n","08:00: average of comments and points per post: 36.84\n","05:00: average of comments and points per post: 36.75\n","04:00: average of comments and points per post: 36.49\n","03:00: average of comments and points per post: 34.19\n","13:00: average of comments and points per post: 33.80\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"gphzh_xqrxEz","colab_type":"text"},"source":["Looking at the average of the number of comments per post and the number of points per post for 'other' posts it is clear that posts published in the evening 6pm - 11pm EEST are most successful. "]},{"cell_type":"markdown","metadata":{"id":"ECrGtutRzeXU","colab_type":"text"},"source":["## Conclusion\n","\n","It stands to reason that `Ask HN` posts receive more comments and less points per post than `Show HN` posts. Questions are likely to attract more comments since by their nature they solicit qualitative interaction; wheras expositions are more likely to attract quantitative kudos than comment engagement. \n","Posts in the `other_posts` subset appear to be more successful in the evening Romanian time which is midday to afternoon time US Eastern Time. It is important to focus on the timezones where Hacker News has the largest number of subscribers to draw the most useful conclusions from this analysis, which is most likely US time: Eastern or Pacific.  \n","The dataset has a limited number of columns to perform interesting analysis which limits the scope of furthering this project unless we find or scrape data with more features but it would be possible to investigate further the profiles of the other type posts: maybe there are posts with certain authors or URLs that receive a high number of comments/points per post. "]}]}