From 0a4e4b3941a266f0547f9c350e82f5de93cbf413 Mon Sep 17 00:00:00 2001
From: qiancao <qiancao227@gmail.com>
Date: Wed, 30 Oct 2024 15:07:35 -0400
Subject: [PATCH] added a few more edits

---
 paper/.paper.md.swp | Bin 28672 -> 28672 bytes
 paper/paper.md      |  33 +++++++++++++--------------------
 2 files changed, 13 insertions(+), 20 deletions(-)

diff --git a/paper/.paper.md.swp b/paper/.paper.md.swp
index 815ac14c3b1e6cbdc0a012c7ff73f43fa7075926..b78ec5096b95b9b8257991525838620b8f95e07a 100644
GIT binary patch
delta 1981
zcmZY9X>1f_9LMozw<onNB^0|@k*6(J0V|Mr&>~Wy*cxbsmYQNJo$gL|htBRSGrMg-
zr4)F>5l!<bf*gkULWxF!AsiA4Z#?k=QZEpQCSC+YLsFE4_}#J%MJM^>ncaEjdj7xp
zues=1bJ0`SI<2v0p?sulhG5GR;`?>Q!Pkz6clRF~v9B;EZ&+I5f3`VldS1bz)LTD9
zd2(Ex96iY~Jvly>9FP3xcuMm3u;loUyDhJMczTsrh$Sg1OP-#wHbaPRjO9}I;ZYu7
z7P>ffH6jS35lhgJvTt{_SFI{;P;ZuwQ7cMrcGbl~x-Ctk)z)l7ccf<Pvd!|x9X%ke
zh%~}cQxEHrm=Os|zcw&0W^f>^$3j-Xkw(Oc=^As}WT&>$kugi^9hw=}V)`BPpJ}L@
zS643mMoe!mlR`YLOIvT#Q*F*rwd1!B(@e*bG2ITkSH@5As?uPtnlfo>f!{X5MnoI9
zXH>V_EIX`4{JI;Ov?HUuKU;{0b66f~F}O+}=L+!=4q!9KHo-v^<{>fvp&WO2-rw2o
z)tR4oyR?81`CP-7s0$YwEZ5cHLVSZhoI(XBmLng<Sweix9s6(+uVM=xM+wq!DN~3u
zIDuZgk7|lkg%QZbu2j(_UM46VXu?dCVj?CW7yVT3SM=c!c3}s$qZ<$6Zd@lA=kXPe
z;~)-TFW$pBUhphV;5Z85^V}mu0XxfZ=QeJfG9vw+1xmACcSkSUnWBDdEUxuy5egTz
zb%#*Mu;C01MJ!}n@nFb`$0V_mwiXHMiDX1=t5s_?Ok<_wSTr7U%47o*40W`z$o;nA
z2alW4)SarfEc5#sLI$aFBnfPbo3d5+YwXHSwV`=ajvN{zH~-nSo^;XD(jrRL_{zxx
zl^n@SJdk)7Dmf?lLg$FdS)9Tfu(1MlD8y*^aQZGGT)cwyh$0LVi&2T`C_)ZCBL$!0
zLo{Sj$$EBXp#q60RN^{i-%P1Dp&RvBjBH%yTm2Hf*n=Ltgcq?CHJFL97=zL9p#axN
z#&W72#7(Mu!+Wm~Z}H~45Qd34NUTPgr)SY9x303no5H%Q-pOOLG?TB*iLumberbvN
zeCa54X>Pvf{{8d*@TOUbuHaso|GZ~f3k6<fMf4WgX_)5ik2_J|p|z*X8XZGd8>@?y
zoSj<VW|^kdndlo%#E3@eCjmWd(JpLygCj#$XCfq&mR4dL{#&hwEBfOm69RM=!eYy)
z=5N=6dbubmDN)Z&E>btD+Xr5vtKFC8H#NsmJ8N28&y#&#^~FM;`}=|p&&1mg9W*+)
zS>j=>=s**pK@GH;sA<JiPi@smzZGdS0y<5FThf5&Ug|<^Q-&5{9;;JRceLvoNy=uC
z6R8?(*K6nrFQ5sHI7h?z4nZ_yDyAS4ztJ;JVn6oc9lVWg_>1KIiC=IShp-;&(2UvH
zBulcBi)>_~pZHzCdGwRE%jm^1Y{Uk%V*$!gh)krzi!&tiG>+i{2|bSw@H46W3EQz1
zCP`~gBU$zAOk@QnAh8m4sKr8W&+aSkky%fqxHtXlQ&msMH{w=OS`!^U7PljMK<x?j
zSJXtL76=%FMTlCgwj_Z@cpwktXK0*;^!lk%byxV*VPk9l-2d0gahsPP^kfZm0U_Dc
TQ%1g$i%V4Q{6cqydEN6bm)6o(

delta 2408
zcma*pdu$X%9Ki9p>w~shZM8zBNu62|inXDLNJN7asMOFB9#$}>cJFTPZeeft*xhR>
z2Cf1LiVv)-Mk9*G_gmbEAsRJmh(==!;v+GdCK?kIgE3Kz2Jk!A1B(9RC7<l$W@mmk
z^PAt^@MvUsG|~{8(c9H6=d{ffd}@R!@4GOyaj$rJ(`d!+hU%KiQhlFKN6=m~X<6CW
zg}9=494;Os#bbH#xUhJ9<6qyG6tB-L9{&y>s#$w>`C*<v9u2DG{+U}R2(blCbS2Tr
zkc-g5qtj52NxB;xkJ6@O_v2P1=xtuvKGtV*vs|t$(}<at7<X>^r(X@pg=Tntb2gIi
zc06BNX3RBQQ+YD!xH4`Ou6@(7rQs^>#dDr%rzF=WzvI$RE17W;%95t-`N~L0Cn+<^
z@N%wht-rR^0AF+a<a6{fX!>b6z<5s1o5%ycYsS4c*-x7dj;;Eo=>_c<&8#0Vy&R!z
zFmxnsLfoLFtCGqM7EgJ&ETNL79Uhu;NwkE4f{D}S*7}T^GrhDPJnIZ9H%w34Iid5(
zN+EuQUf&Zle}I$Eu@|?VA;d5;SX3p%LbT!-&m1j2d$=-;ls;V<UUBtD(NgAhL2&E5
z^TH{ywvxP&5Q3Eu7warwh>o#^5MR>u7x);Dz`|N|;XIs+bMOhnzJwjvgf&PZi6xke
zOVEG`*hgp|;9c}oiM;6MWHv5G6Y)xXRUyQ4<j=Es6j?ZEK@|>>Q{P}8UdD@f0fR`Q
z7gym*%*N%Ig<5PU_qL%C4JA{B_>f=zHtxcm=*LpbLt$3*j09gUogQ>uUmIStY<e{K
zb7kXXmRTCo^I2ZcH(7k?D{_$x46N%~f=^e}MP&H>io?<H>)u-;g>=fgPRcbhWROK`
zR~f!EeQD@^+Jh~tnx=R8WST8wO2(Z`Hs`AZxj8PgXqt2^%Ng{7KUR&@ok}^w@*J7g
zJq6oWH<k^i&3HO^W_5qmPlshUeif}gv8zm<y%YX&)5b{qppr%+Q3%Y*7Ius8$gJh~
zV;Qbom(3J2Tu*Y}OFM(XNZ+E$ledF|eXDCdGn2Cn9cRMu4Nv(&*UfL{B@<T9?CE?u
zQEZN#EyQ2At(xhF0f!}T#)YWI4=nxn*nupTV-9pmRO1Xxz;~p;x7dT%@fvmR72JaL
zrQ}ovYkQ2W9>x230s%&F57xuLa?C>$Do~Cx{7mq>FpL-$<4UwZq7jpEKJ*Wrgj1i9
zm+He;J9d_YZ+0G!lvBByWmoW4XH!H3*LF_Lr_2E*hn$?w9%Xw<*B4!E$ec`}1exBr
za?>W~OV4t$N~UzeD3`XRNIQv<AnV9EeZw#GP^@O2bOw|gckHB@P&NtX%e0wFOVTf9
z#PX(P`npycc0%Ut#J^gST*ZdPcCCem17=)Fdi4wfvKgF6;wtN(9@UKh5mo*~SY&oe
z*~&Gn{|?dPu%*=4FQW!C;Mg3E%>+BIy%3e2wo{>Sdq(THiOBGyeQA@>dP5c83M!vq
zI8QEc)=%|L-YvW<Y{MGz=p<>FRw3!*M%K^q2JxH>u?%IIH2sP=ZGBn^Aa3Onx@xb>
z83~tq5SJ>MRB@l?u5<{}B?|je%e0l@mKSEfE!?|!b7|y;U{CiQVNcKINHDiMo)`W7
z{h~$an!KMhJVqMpJ+!cxTbHtbT2YG{SZ4`g;vjqJN4$r3um~OajQYF}_v1dS!*a|*
z1I~x8<#o{YJg*D7e#xw+P_IG{E<huGW`q6&UA&*dHss(zZ{$AcTD}C2u}L353Q1gs
znW)3>)OKCpKfrFhf)}u<LR=;`a+1b&{zTl4EJi5WccT*vFo|705qfrx@(1ETG+d#!
zmQJOV77DAYSTtV!)Mhsq*9(}Cb}o}0;@^O44BfltNbs9gA6&Qg!Rijj@o8mbvn*fu
X$J#xS@cB${G^n3{Bw__M9lQPkc4t^9

diff --git a/paper/paper.md b/paper/paper.md
index bc4418e..1dabf7d 100644
--- a/paper/paper.md
+++ b/paper/paper.md
@@ -47,19 +47,19 @@ bibliography: paper.bib
 `calzone` is a Python package for evaluating the calibration of probabilistic outputs of classifier models. It provides a set of functions and classes for visualizing calibration and computing calibration metrics given a representative dataset with the model's predictions and true class labels. The metrics provided in `calzone` include: Expected Calibration Error (ECE), Maximum Calibration Error (MCE), Hosmer-Lemeshow (HL) statistic, Integrated Calibration Index (ICI), Spiegelhalter's Z-statistics and Cox's calibration slope/intercept. The package is designed with versatility in mind. For many of the metrics, users can adjust the binning scheme and toggle between top-class or class-wise calculations. 
 
 # Statement of need
-Classification is one of the most fundamental tasks in machine learning. Classification models are often evaluated by a proper scoring rule, such as the cross-entropy or mean square error. Examination of the distinguishing power (resolution), such as AUC or Se/Sp are also used to evaluate the model performance. However, the reliability or calibration performance of the model is often overlooked. 
+Classification is one of the most fundamental tasks in machine learning. Classification models are often evaluated by a proper scoring rule, such as cross-entropy or mean square error. Examination of the discrimination performance (resolution), such as AUC or Se/Sp are also used to evaluate the model performance. However, the reliability or calibration performance of the model is often overlooked. 
 
 @Brocker_decompose has shown that the proper scoring rule can be decomposed into the resolution and reliability. That means even if the model has high resolution (high AUC), it may not be a reliable or calibrated model. In many high-risk machine learning applications, such as medical diagnosis, the reliability of the model is of paramount importance. 
 
-We refer calibration as the agreement between the predicted probability and the true posterior probability of a class-of-interest, $P(D=1|\hat{p}=p) = p$. This is defined as moderate calibration by @Calster_weak_cal .
+We refer to calibration as the agreement between the predicted probability and the true posterior probability of a class-of-interest, $P(D=1|\hat{p}=p) = p$. This is also termed as moderate calibration by @Calster_weak_cal .
 
-In the `calzone` package, we provide a set of functions and classes for calibration visualization and metrics computation. Existing libraries such as `scikit-learn` are often not dedicated to calibration metrics computation and don't provide calibration metrics computation that are widely used in the statistical literature. Most libraries for calibration are focusing on calibrating the model instead of measuring the level of calibration with various metrics. `calzone` is dedicated to calibration metrics computation and visualization.
+In the `calzone` package, we provide a set of functions and classes for calibration visualization and metrics computation. Existing libraries such as `scikit-learn` are often not dedicated to calibration metrics computation and don't provide calibration metrics computation that are widely used in the statistical literature. Other libraries are focused on implementing calibration methods instead of ways to evaluate calibration [TODO: cite]. 
 
 # Functionality
 
 ## Reliability Diagram
 
-Reliability Diagram is a graphical representation of the calibration of a classification model [@Brocker_reldia]. It groups the predicted probabilities into bins and plots the mean predicted probability against the empirical frequency in each bin. The reliability diagram can be used to assess the calibration of the model and to identify any systematic errors in the predictions. In addition, we add the option to plot with error bars to show the confidence interval of the empirical frequency in each bin. The error bars are calculated using Wilson's score interval [@wilson_interval]. We provide an example simulated dataset in the `example_data` folder using beta-binomial distribution [@beta-binomial]. Users can generate simulated data using the `fake_binary_data_generator` class in the `utils` module.
+The reliability diagram is a graphical representation of the calibration of a classification model [@Brocker_reldia]. It groups the predicted probabilities into bins and plots the mean predicted probability against the empirical frequency in each bin. The reliability diagram can be used to assess the calibration of the model and to identify any systematic errors in the predictions. In addition, `calzone` gives the option to also plot the confidence interval of the empirical frequency in each bin. The confidence intervals are calculated using Wilson's score interval [@wilson_interval]. We provide an example analsis in the `example_data` folder using beta-binomial distribution [@beta-binomial]. Users can generate simulated data using the `fake_binary_data_generator` class in the `utils` module.
 
 ```python
 from calzone.utils import reliability_diagram
@@ -88,10 +88,10 @@ plot_reliability_diagram(
 
 ## Calibration metrics
 
-`calzone` provides functions to compute various calibration metrics. `calzone` also has a `CalibrationMetrics()` class which allows the user to compute the calibration metrics in a more convenient way. The following are the metrics that are currently supported in `calzone`: 
+`calzone` provides functions to compute various calibration metrics. The `CalibrationMetrics()` class allows the user to compute the calibration metrics in a more convenient way. The following are metrics that are currently supported in `calzone`: 
 
 ### Expected Calibration Error (ECE) and Maximum Calibration Error (MCE)
-Expected Calibration Error (ECE), Maximum Calibration Error (MCE) and binning-based methods [@guo_calibration;@Naeini_ece] aim to measure the average deviation between predicted probability and true probability. We provide the option to use equal-width binning or equal-count binning, labeled as ECE-H and ECE-C respectively. Users can also choose to compute the metrics for the class-of-interest or the top-class. In the case of class-of-interest, the program will treat it as a 1-vs-rest classification problem. It can be computed in `calzone` as follows:
+Expected Calibration Error (ECE), Maximum Calibration Error (MCE) and other binning-based methods [@guo_calibration;@Naeini_ece] aim to measure the average deviation between predicted probability and true probability. We provide the option to use equal-width binning or equal-count binning, labeled as ECE-H and ECE-C respectively. Users can also choose to compute the metrics for the class-of-interest or the top-class. In the case of class-of-interest, `calzone` will evaluate the calibration of a one-vs-rest classification problem. The following snipped demonstrates how these metrics are calculated in our package:
 
 ```python
 from calzone.metrics import calculate_ece_mce
@@ -113,7 +113,7 @@ ece_h_classone, mce_h_classone = calculate_ece_mce(
 
 
 ### Hosmer-Lemeshow statistic (HL)
-Hosmer-Lemeshow statistic (HL) is a statistical test for the calibration of a probabilistic model. It is a chi-square based test that compares the observed and expected number of events in each bin. The null hypothesis is that the model is well calibrated. HL-test first bins data into predicted probability bins (equal-width $H$ or equal-count $C$) and the test statistic is calculated as:
+The Hosmer-Lemeshow (HL) statistical test is for evaluating the calibration of a probabilistic model. It is a chi-square-based test that compares the observed and expected number of events in each bin. The null hypothesis is that the model is well calibrated. HL-test first bins data into predicted probability bins (equal-width $H$ or equal-count $C$) and the test statistic is calculated as:
 $$
 \text{HL} = \sum_{m=1}^{M} \frac{(O_{1,m}-E_{1,m})^2}{E_{1,m}(1-\frac{E_{1,m}}{N_m})}  \sim \chi^2_{M-2}
 $$
@@ -127,7 +127,7 @@ HL_H_ts, HL_H_p, df = hosmer_lemeshow_test(
     bin_count=bin_counts
 )
 ```
-When performing the HL test on validation sets that are not used in training, the degree of freedom of the HL test changes from $M-2$ to $M$. Intuitively, $\frac{(O_{1,m}-E_{1,m})^2}{E_{1,m}(1-\frac{E_{1,m}}{N_m})}$ is the difference squared divided by the variance of a binomial distribution and follows a chi-square distribution with 1 degree of freedom. Hence, the sum of $M$ chi-square distributions with 1 degree of freedom is a chi-square distribution with $M$ degrees of freedom if the data has no effect on the model. In `calzone`, user can sepecify the degree of freedom of the HL test by setting the `df` parameter.
+In `calzone`, user can sepecify the degree of freedom of the HL test by setting the `df` parameter.This is useful because when performing the HL test on validation sets that are not used in training, the degree of freedom of the HL test changes from $M-2$ to $M$ [TODO: cite]. Intuitively, $\frac{(O_{1,m}-E_{1,m})^2}{E_{1,m}(1-\frac{E_{1,m}}{N_m})}$ is the difference squared divided by the variance of a binomial distribution and follows a chi-square distribution with 1 degree of freedom. Hence, the sum of $M$ chi-square distributions with 1 degree of freedom is a chi-square distribution with $M$ degrees of freedom if the data has no effect on the model. 
 
 ### Cox's calibration slope/intercept
 Cox's calibration slope/intercept is a non-parametric method for assessing the calibration of a probabilistic model [@Cox]. A new logistic regression model is fitted to the data, with the predicted odds ($\frac{p}{1-p}$) as the dependent variable and the true probability as the independent variable. The slope and intercept of the regression line are then used to assess the calibration of the model. A slope of 1 and intercept of 0 indicates perfect calibration. To test whether the model is calibrated, fix the slope to 1 and fit the intercept. If the intercept is significantly different from 0, the model is not calibrated. Then, fix the intercept to 0 and fit the slope. If the slope is significantly different from 1, the model is not calibrated.
@@ -144,11 +144,7 @@ cox_slope, cox_intercept, cox_slope_ci, cox_intercept_ci = cox_regression_analys
     fix_slope=True
 )
 ```
-The values of the slope and intercept give you a sense of the form of miscalibration. A slope greater than 1 indicates that the model is overconfident at high probabilities and underconfident at low probabilities, and vice versa. An intercept greater than 0 indicates that the model is overconfident in general, and vice versa. Notice that even if the slope is 1 and the intercept is 0, the model might not be calibrated, as Cox's calibration analysis fails to capture some types of miscalibration, including quadratic effects or other non-linearities.
-
-### Integrated calibration index (ICI)
-
-The integrated calibration index (ICI) is very similar to Expected calibration error (ECE). It also tries to measure the average deviation between predicted probability and true probability. However, ICI does not use binning to estimate the true probability of a group of samples with similar predicted probability. Instead, ICI uses curve smoothing techniques to fit the regression curve and uses the regression result as the true probability [@ICI_austin]. The ICI is then calculated using the following formula:
+The values of the slope and intercept can represent miscalibration throughout the range of probabiliy outputs. The integrated calibration index (ICI) is very similar to Expected calibration error (ECE). It also tries to measure the average deviation between predicted probability and true probability. However, ICI does not use binning to estimate the true probability of a group of samples with similar predicted probability. Instead, ICI uses curve smoothing techniques to fit the regression curve and uses the regression result as the true probability [@ICI_austin]. The ICI is then calculated using the following formula:
 $$
 \text{ICI} = \frac{1}{n}\sum_{i=1}^{n} |f(p_i)-p_i|
 $$
@@ -179,8 +175,7 @@ loess_ici, lowess_fit_p, lowess_fit_p_correct = lowess_regression_analysis(
 )
 ```
 
-
-Notice that flexible curve fitting methods such as loess regression are very sensitive to the choice of span and delta parameters. The user can visualize the fitting result to avoid overfitting or underfitting.
+Notice that flexible curve fitting methods such as Loess regression are very sensitive to the choice of span and delta parameters. The user can visualize the fitting result to avoid overfitting or underfitting.
 
 ### Spiegelhalter's Z-test
 
@@ -203,8 +198,6 @@ z, p_value = spiegelhalter_z_test(
 )
 ```
 
-
-
 ### Metrics class
 `calzone` also provides a class called `CalibrationMetrics()` to calculate all the metrics mentioned above. The user can also use this class to calculate the metrics.
 
@@ -220,9 +213,9 @@ CalibrationMetrics.calculate_metrics(
 )
 ```
 # Other features
-## Bootstrapping
+## Confidence intervals
 
-`calzone` also provides bootstrapping to calculate the confidence intervals of the metrics. The user can specify the number of bootstrap samples and the confidence level. 
+In addition to point estimates of calibration performance, `calzone` also provides bootstrapping to calculate the confidence intervals of the metrics. The user can specify the number of bootstrap samples and the confidence level. 
 ```python
 from calzone.metrics import CalibrationMetrics
 
@@ -235,7 +228,7 @@ CalibrationMetrics.bootstrap(
     n_samples=1000
 )
 ```
-and it will return a structured numpy array.
+and a structured numpy array will be returned.
 
 ## Subgroup analysis
 `calzone` will perform subgroup analysis by default in the command line user interface. If the user input CSV file contains a subgroup column, the program will compute metrics for the entire dataset and for each subgroup.