Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ptime mapper script #737

Draft
wants to merge 80 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
a77a77d
Create migration to add ptime id to person model
RandomTannenbaum Jun 21, 2024
665bf0b
Start writing mapper script
RandomTannenbaum Jun 21, 2024
2f1660f
Implement basic PTime client class and install webmock gem
ManuelMoeri Jun 24, 2024
0555ec7
Create new migration to fix naming of ptime_emplyoee_id attribute
ManuelMoeri Jun 24, 2024
83fa511
Add first test to see how webmock works
ManuelMoeri Jun 24, 2024
fbc8987
Fix first client spec
RandomTannenbaum Jun 24, 2024
6a75e33
Write json body to test employee endpoint
RandomTannenbaum Jun 25, 2024
6a9a89a
Fix tests by allowing localhost connection and write negative test fo…
RandomTannenbaum Jun 25, 2024
9ddd723
Make ptime client specs not read values from env variables
RandomTannenbaum Jun 25, 2024
7ab6727
Set error messages on errors in client and change test data urls to e…
RandomTannenbaum Jun 25, 2024
8f85e91
Fix params of ptime client and implement basic mapper script
RandomTannenbaum Jun 25, 2024
f945378
Add new tests and remove unnecessary
RandomTannenbaum Jun 25, 2024
933c2fc
Add test for ambiguous matched people
ManuelMoeri Jun 26, 2024
ff37bcc
Add email to filter before matching as well
ManuelMoeri Jun 26, 2024
244369f
Possibly make rubocop happy
ManuelMoeri Jun 26, 2024
7e58084
Try to make rubocop happy again
ManuelMoeri Jun 26, 2024
132424a
Add evaluate task for mapping script
RandomTannenbaum Jun 27, 2024
2070ffb
Outsource mapper script code to methods and fix tests
RandomTannenbaum Jun 27, 2024
b89c933
Add test for dry run
RandomTannenbaum Jun 27, 2024
b44a2d0
Make rubocop happy :)
RandomTannenbaum Jun 27, 2024
f9be66c
Remove unnecessary requires
RandomTannenbaum Jun 27, 2024
104185e
Replace .count == 1 with .one?
RandomTannenbaum Jun 27, 2024
aa4c5f5
Start writing script that updates the data of the people in skills
RandomTannenbaum Jun 27, 2024
7d775c3
Write script that maps the data from a ptime employee to the correspo…
RandomTannenbaum Jun 28, 2024
82d03f0
Disable rubocop metrics all together instead of one by one in assign …
RandomTannenbaum Jun 28, 2024
96913d2
Fix file naming of spec files
RandomTannenbaum Jun 28, 2024
907da90
Move class into module and fix mapping of person to employee data, wr…
RandomTannenbaum Jun 28, 2024
35f4b0d
Fix updating of people data and test it and fix faulty key in json us…
RandomTannenbaum Jun 28, 2024
1190569
Fix update people data specs by assigning the ptime base url env var …
RandomTannenbaum Jun 28, 2024
5ac5585
Only update ptime employee id if its not already set
RandomTannenbaum Jun 28, 2024
8c8e0e4
Test if marital status can be changed in specs
RandomTannenbaum Jun 28, 2024
dd0bacd
Write rake task that runs the people update
RandomTannenbaum Jun 28, 2024
b2d5d0b
Show message in assign employee ids script conditionally
RandomTannenbaum Jun 28, 2024
dbe7132
Add delayed jobs and daemon gems and start writing delayed jobs
RandomTannenbaum Jul 1, 2024
6fb4df1
Fix specs to work with delayed job
RandomTannenbaum Jul 2, 2024
480e328
Remove delayed job functionality
RandomTannenbaum Jul 3, 2024
138fab7
Add first approach for fetching people from ptime API
ManuelMoeri Jul 4, 2024
7f6abdd
Add helper and write test for fetching employee data
ManuelMoeri Jul 4, 2024
ea3d4e5
Finish test and fix logic in displaying people
ManuelMoeri Jul 5, 2024
3a6b7c0
Implement correct routing of person and fix a problem in the docker s…
ManuelMoeri Jul 5, 2024
c25e18c
Possibly fix docker compose file
ManuelMoeri Jul 8, 2024
4385c4a
Fix tests and implement logic for sorting people
ManuelMoeri Jul 8, 2024
202006a
Implement logic for fallback and cleanup helper method
ManuelMoeri Jul 9, 2024
1dc3065
Improve fallback logic by making it more readable and shorter
ManuelMoeri Jul 9, 2024
1267736
Implement first approach on calling the ptime api after 5 minutes
ManuelMoeri Jul 10, 2024
512593b
Improve logic of 5 minutes request and change client.rb
ManuelMoeri Jul 11, 2024
c9e9825
Add logic to load site correctly and remove unnecessary code
ManuelMoeri Jul 12, 2024
39acf6e
Fix logic of rendering people and cleanup code
ManuelMoeri Jul 15, 2024
d967660
Add tests for person_helper
ManuelMoeri Jul 17, 2024
c657e8f
Add ptime_helpers to cleanup code
ManuelMoeri Jul 17, 2024
bcb3ebe
Fix tests
ManuelMoeri Jul 17, 2024
456f27b
Fix last test and remove pry
ManuelMoeri Jul 17, 2024
4718e92
Fix test and add comment
ManuelMoeri Jul 18, 2024
bf771aa
Generalize window resizing of capybara
ManuelMoeri Jul 18, 2024
189ab80
Cleanup code and add comments
ManuelMoeri Jul 19, 2024
2e2ae6b
Add one more test for client.rb
ManuelMoeri Jul 19, 2024
766e942
Reverse test from last commit
ManuelMoeri Jul 19, 2024
aa36e82
change location of helper
kcinay055679 Jul 25, 2024
4153a5b
set ptime stubing global
kcinay055679 Jul 25, 2024
87d84d8
clean up ptime client specs
kcinay055679 Jul 25, 2024
96391b2
create json file for fixtures
kcinay055679 Jul 25, 2024
cce291b
use hashes instead of strings for json keys create global ptime stub …
kcinay055679 Jul 25, 2024
d3bb9c3
use custom ptime stubbing method everywhere
kcinay055679 Jul 25, 2024
dd96900
fix tests after always using [:data]
kcinay055679 Jul 25, 2024
0d1c4ff
clean up ptime client
kcinay055679 Jul 25, 2024
7550234
improve structure of people search
kcinay055679 Jul 25, 2024
552cf7d
Client isn't longer controlled by random env vars
kcinay055679 Jul 25, 2024
d9c02be
simplify ptime_helpers
kcinay055679 Jul 25, 2024
44d76bd
clean up PR
kcinay055679 Jul 26, 2024
71db98a
Clean up Skills db fallback logic
kcinay055679 Jul 26, 2024
3a7997a
improve readability by using more helper
kcinay055679 Jul 26, 2024
d19bd8e
clean up
kcinay055679 Jul 26, 2024
57ad58a
add client test
kcinay055679 Jul 29, 2024
5411381
Merge pull request #748 from puzzle/feature/745-fetch-people-search-e…
kcinay055679 Jul 29, 2024
af3def7
ds muäs so
kcinay055679 Jul 29, 2024
2763d61
so its aber
kcinay055679 Jul 29, 2024
d272a03
Resolve conversations
RandomTannenbaum Jul 29, 2024
48fb6be
Remove unneeded spec and fix spec that tests the not found case when …
RandomTannenbaum Jul 30, 2024
a52ceaa
Fix order of name interpolation in id mapper script
RandomTannenbaum Jul 30, 2024
65fb684
Update json fixtures and use full_name instead of interpolating it
RandomTannenbaum Jul 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -99,3 +99,5 @@ config/docker/development/home/rails/.bash_history

/app/assets/builds/*
!/app/assets/builds/.keep

.sec
1 change: 1 addition & 0 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ group :test do
gem 'capybara'
gem 'selenium-webdriver', '>= 4.11.8'
gem 'simplecov'
gem 'webmock'
# Use fixed version of webdrivers to avoid compatibility issues with chrome and chromedriver
end
# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
Expand Down
10 changes: 10 additions & 0 deletions Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ GEM
ast (2.4.2)
base64 (0.2.0)
bcrypt (3.1.20)
bigdecimal (3.1.8)
bindata (2.5.0)
bleib (0.0.8)
bootsnap (1.18.3)
Expand Down Expand Up @@ -118,6 +119,9 @@ GEM
deep_merge (~> 1.2, >= 1.2.1)
countries (6.0.0)
unaccent (~> 0.3)
crack (1.0.0)
RandomTannenbaum marked this conversation as resolved.
Show resolved Hide resolved
bigdecimal
rexml
crass (1.0.6)
cssbundling-rails (1.4.0)
railties (>= 6.0.0)
Expand Down Expand Up @@ -166,6 +170,7 @@ GEM
rainbow
rubocop (>= 1.0)
sysexits (~> 1.1)
hashdiff (1.1.0)
hashie (5.0.0)
http-accept (1.7.0)
http-cookie (1.0.5)
Expand Down Expand Up @@ -444,6 +449,10 @@ GEM
version_gem (1.1.4)
warden (1.2.9)
rack (>= 2.0.9)
webmock (3.23.1)
addressable (>= 2.8.0)
crack (>= 0.3.2)
hashdiff (>= 0.4.0, < 2.0.0)
websocket (1.2.10)
websocket-driver (0.7.6)
websocket-extensions (>= 0.1.0)
Expand Down Expand Up @@ -517,6 +526,7 @@ DEPENDENCIES
stimulus-rails
turbo-rails
tzinfo-data
webmock

BUNDLED WITH
2.4.10
60 changes: 60 additions & 0 deletions app/domain/ptime/assign_employee_ids.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# This script will assign each person the corresponding employee ID from PuzzleTime
module Ptime
class AssignEmployeeIds

# rubocop:disable Rails/Output
def run(should_map: false)
puts 'Notice this is a dry run and mapping will not happen!' unless should_map
fetch_data

puts "Currently there are:
- #{@ptime_employees.length} employees in PuzzleTime
- #{@skills_people.count} people in PuzzleSkills
This is a difference of #{(@skills_people.count - @ptime_employees.length).abs} entries"
map_employees(should_map)
end

private

# rubocop:disable Metrics
def map_employees(should_map)
puts 'Assigning employee IDs now...' if should_map

unmatched_entries = []
mapped_people_count = 0

@ptime_employees.each do |ptime_employee|
ptime_employee_name = ptime_employee[:attributes][:full_name]
ptime_employee_email = ptime_employee[:attributes][:email]
matched_person = Person.where(ptime_employee_id: nil).find_by(email: ptime_employee_email)

if matched_person.nil?
unmatched_entries << { name: ptime_employee_name, id: ptime_employee[:id] }
else
if should_map
matched_person.ptime_employee_id = ptime_employee[:id]
matched_person.save!
end
mapped_people_count += 1
end
end

puts '--------------------------'
puts "#{mapped_people_count} people were matched successfully"
puts '--------------------------'
puts "#{unmatched_entries.size} people didn't match"
unmatched_entries.each { |entry| puts "- #{entry[:name]} with id #{entry[:id]}" }
end
# rubocop:enable Metrics

MAX_NUMBER_OF_FETCHED_EMPLOYEES = 1000
def fetch_data
puts 'Fetching required data...'
@ptime_employees = Ptime::Client.new.request(:get, 'employees',
{ per_page: MAX_NUMBER_OF_FETCHED_EMPLOYEES })
@skills_people = Person.all
puts 'Successfully fetched data'
end
# rubocop:enable Rails/Output
end
end
49 changes: 49 additions & 0 deletions app/domain/ptime/client.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# frozen_string_literal: true

require 'rest_client'
require 'base64'

module Ptime
class Client
def initialize
@base_url = "#{ENV.fetch('PTIME_BASE_URL')}/api/v1/"
end

def request(method, endpoint, params = {})
url = @base_url + endpoint

if last_ptime_error_more_than_5_minutes_ago?
send_request_and_parse_response(method, url, params)
else
raise CustomExceptions::PTimeClientError, 'Error'
end
end

private

def last_ptime_error_more_than_5_minutes_ago?
last_request_time = ENV.fetch('LAST_PTIME_ERROR', nil)
return true if last_request_time.nil?

last_request_time.to_datetime <= 5.minutes.ago
end

def ptime_request(method, url)
RestClient::Request.new(
:method => method,
:url => url,
:user => ENV.fetch('PTIME_API_USERNAME'),
:password => ENV.fetch('PTIME_API_PASSWORD'),
:headers => { :accept => :json, :content_type => :json }
)
end

def send_request_and_parse_response(method, url, params)
url += "?#{params.to_query}" if method == :get && params.present?
response = ptime_request(method, url).execute
JSON.parse(response.body, symbolize_names: true)[:data]
rescue RestClient::ExceptionWithResponse
raise CustomExceptions::PTimeClientError, 'Error'
end
end
end
35 changes: 35 additions & 0 deletions app/domain/ptime/update_people_data.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# frozen_string_literal: true

module Ptime
class UpdatePeopleData
ATTRIBUTE_MAPPING = { shortname: :shortname, email: :email, marital_status: :marital_status,
graduation: :title }.freeze

# rubocop:disable Metrics
def run
ptime_employees = Ptime::Client.new.request(:get, 'employees', { per_page: 1000 })
ptime_employees.each do |ptime_employee|
ptime_employee_firstname = ptime_employee[:attributes][:firstname]
ptime_employee_lastname = ptime_employee[:attributes][:lastname]
ptime_employee_name = "#{ptime_employee_firstname} #{ptime_employee_lastname}"

skills_person = Person.find_by(ptime_employee_id: ptime_employee[:id])
skills_person ||= Person.new

skills_person.name = ptime_employee_name
skills_person.ptime_employee_id ||= ptime_employee[:id]
ptime_employee[:attributes].each do |key, value|
if key.to_sym.in?(ATTRIBUTE_MAPPING.keys)
skills_person[ATTRIBUTE_MAPPING[key.to_sym]] = value
end
end
skills_person.company = Company.first
skills_person.birthdate = '1.1.2000'
skills_person.location = 'Bern'
skills_person.nationality = 'CH'
skills_person.save!
end
end
# rubocop:enable Metrics
end
end
5 changes: 5 additions & 0 deletions app/exceptions/custom_exceptions.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
module CustomExceptions

class PTimeClientError < StandardError; end

end
33 changes: 33 additions & 0 deletions app/helpers/person_helper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -76,4 +76,37 @@ def not_rated_default_skills(person)
certificate: false, core_competence: false })
end
end

def sorted_people
fetch_ptime_or_skills_data.sort_by { |e| e.first.downcase }
end

def fetch_ptime_or_skills_data
all_skills_people = Person.all.map { |p| [p.name, person_path(p)] }
return all_skills_people unless Skills.ptime_available?

ptime_employees = Ptime::Client.new.request(:get, 'employees', { per_page: 1000 })
build_dropdown_data(ptime_employees)
rescue CustomExceptions::PTimeClientError
ENV['LAST_PTIME_ERROR'] = DateTime.current.to_s
all_skills_people
end

def build_dropdown_data(ptime_employees)
ptime_employees.map do |ptime_employee|
ptime_employee_name = append_ptime_employee_name(ptime_employee)
skills_person = Person.find_by(ptime_employee_id: ptime_employee[:id])
ptime_employee_id = ptime_employee[:id]
already_exists = ptime_employee_id.in?(Person.pluck(:ptime_employee_id))
path = new_person_path(ptime_employee_id: ptime_employee_id)
path = person_path(skills_person) if already_exists

[ptime_employee_name, path]
end
end

# Once https://github.com/puzzle/skills/issues/744 is merged there should be no need for this
def append_ptime_employee_name(ptime_employee)
"#{ptime_employee[:attributes][:firstname]} #{ptime_employee[:attributes][:lastname]}"
end
end
4 changes: 2 additions & 2 deletions app/helpers/select_helper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

module SelectHelper
def select_when_availabale(obj)
selected = obj ? obj.id : ''
prompt = obj ? false : true
selected = obj.present? ? polymorphic_path(obj) : ''
prompt = obj.nil?
{ selected: selected, prompt: prompt, disabled: '' }
end

Expand Down
2 changes: 1 addition & 1 deletion app/javascript/controllers/dropdown_controller.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,6 @@ export default class extends Controller {
}

handleChange(event) {
window.location.href = event.target.dataset.value + event.target.value;
window.location.href = event.target.value;
}
}
2 changes: 1 addition & 1 deletion app/views/people/_search.html.haml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
%div.d-flex.align-items-center.justify-content-between
%div.d-flex.col-9.gap-3
%span.col-6{"data-controller": "dropdown"}
= collection_select :person_id, :person, Person.all.sort_by {|p| p.name.downcase }, :id, :name, select_when_availabale(person), {data:{"dropdown-target": "dropdown" , action: "change->dropdown#handleChange", value: "/people/"}}
= select :person_id, :person, options_for_select(sorted_people, select_when_availabale(person)), {prompt: person.nil?}, {data: { "dropdown-target" => "dropdown", action: "change->dropdown#handleChange"}}
%div.d-flex.align-items-center.text-gray
= "#{t 'profile.updated_at'}: #{@person&.last_updated_at.strftime("%d.%m.%Y")}" if @person&.last_updated_at
%a.d-flex.justify-content-between#new-person-button{href: "/people/new"}
Expand Down
4 changes: 4 additions & 0 deletions config/application.rb
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@
Bundler.require(*Rails.groups)

module Skills
def self.ptime_available?
ActiveModel::Type::Boolean.new.cast(ENV.fetch('PTIME_API_ACCESSIBLE', true))
end

class Application < Rails::Application
# Initialize configuration defaults for originally generated Rails version.
config.load_defaults 7.0
Expand Down
5 changes: 5 additions & 0 deletions db/migrate/20240624122411_add_ptime_employee_id_to_people.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
class AddPtimeEmployeeIdToPeople < ActiveRecord::Migration[7.0]
def change
add_column :people, :ptime_employee_id, :integer
end
end
3 changes: 2 additions & 1 deletion db/schema.rb
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#
# It's strongly recommended that you check this file into your version control system.

ActiveRecord::Schema[7.0].define(version: 2024_06_03_085509) do
ActiveRecord::Schema[7.0].define(version: 2024_06_24_122411) do
# These are extensions that must be enabled in order to support this database
enable_extension "plpgsql"

Expand Down Expand Up @@ -148,6 +148,7 @@
t.string "email"
t.integer "department_id"
t.string "shortname"
t.integer "ptime_employee_id"
t.index ["company_id"], name: "index_people_on_company_id"
end

Expand Down
6 changes: 4 additions & 2 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ services:
stdin_open: true
environment:
- RAILS_DB_HOST=postgres
env_file: .env
env_file:
- .env
- .sec
build:
context: ./config/docker/development
dockerfile: Rails.dockerfile
Expand Down Expand Up @@ -39,7 +41,7 @@ services:
/bin/bash -c "
curl -fsSL https://deb.nodesource.com/setup_18.x | bash - &&
apt-get install -y nodejs &&
npm install -g yarn && bin/assets &&
npm install -g yarn && yarn add nodemon esbuild && bin/assets &&
sleep infinity"
volumes:
- ./:/myapp
Expand Down
16 changes: 16 additions & 0 deletions lib/tasks/ptime.rake
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
namespace :ptime do
desc 'assign puzzletime employee ids to people'
task :assign => :environment do
Ptime::AssignEmployeeIds.new.run(should_map: true)
end

desc 'evaluate assignment of employee ids to people'
task :evaluate_assign => :environment do
Ptime::AssignEmployeeIds.new.run
end

desc 'update person data with the data from ptime'
task :update_people => :environment do
Ptime::UpdatePeopleData.new.run
end
end
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
"autoprefixer": "^10.4.17",
"bootstrap": "^5.3.2",
"bootstrap-icons": "^1.11.3",
"esbuild": "^0.20.0",
"nodemon": "^3.0.3",
"esbuild": "^0.23.0",
"nodemon": "^3.1.4",
"postcss": "^8.4.33",
"postcss-cli": "^11.0.0",
"sass": "^1.70.0",
Expand Down
Loading