TF2: Getting CAM w/ new CLUBB external to match old CAM-CLUBB results

[bstephens82]

@vlarson suggested it would be useful to recover old CAM results with the new CLUBB external, as a check to make sure that the new CLUBB code is backward compatible. Since Kate's pull request is using an 053 tag, I thought the way to proceed would be to clone a clean 053 tag and run it, then try and reproduce those results with Kate's new PR code.

So far the two code bases shows pretty different climates from 14-month runs using the following create_newcase command:

./create_newcase --case ${casename} --compset ${compset} --res ne30pg3_ne30pg3_mg17 --driver ${driver} --project P93300642 --run-unsupported --pecount 1800 --queue regular --walltime 12:00:00

where compset=FHIST and driver=mct.

I have found a couple of differences between the code bases that I thought might be important, for example the values of omicron and upsilon_precip_frac_rat (Brian says these should have no effect unless SILHS is active), and also in Kate's code the PDF call placement was in the "post" position while in the clean 053 tag it was in the "pre" position, so I've reverted those things and I still find pretty significant disagreement. I've gone through all other parameters and flags and am not aware of any other disagreements, so my next guess is structural changes to CLUBB's code, which may be harder to track down.

Here is SWCF as an example of the differences in climate.

From old CAM 053 tag:

SWCF from Kate's PR code:

[vlarson]

For UWM's reference (@bmg929), UWM's private discussion concerning this merge is at https://github.com/larson-group/cam/issues/140.

I wonder if this older larson-group/cam tag

https://github.com/larson-group/cam/releases/tag/cam_12a6970_clubb_335d83b_20211111

based on the 031 trunk would reproduce either Kate's new branch or else the 053 trunk.

Likewise, does that older larson-group/cam tag match the newest version of larson-group/cam branch clubb_silhs_devel?

[bstephens82]

The versions of CLUBB in these two repos are

Clean 053 CAM tag: Feb 20, 2020, commit b0575cc936312f99dcffe9f1f7bea6932cd1fd4a
Kate's PR code: March 10, 2022, commit 9976b866185e562df3c99f7fbe96379192d3ad9d

---

I am currently running some tests comparing Brian Griffin's Nov 11 CAM-CLUBB tag from the Larson-group repo---which is based on a CAM 031 tag---to a clean CAM 031 tag. Brian thought those results were similar. If they are this might help to narrow down where the differences are coming from.

[bstephens82]

I tried this and I don't think the output is very interesting, I think it just reflects some things that Kate has added to the default namelist, and the fact that clubb_up2_vp2_factor was renamed to clubb_up2_sfc_coef. The l_e3sm_config flag shouldn't make any difference since diag_Lscale=F for these runs. The ice_latent flag has been removed from CLUBB in the current version.

>  clubb_c_invrs_tau_bkgnd              =  1.0
>  clubb_c_invrs_tau_n2         =  0.1
>  clubb_c_invrs_tau_n2_clear_wp3               =  0.0
>  clubb_c_invrs_tau_n2_wp2             =  0.2
>  clubb_c_invrs_tau_n2_wpxp            =  0.0
>  clubb_c_invrs_tau_n2_xp2             =  0.2
>  clubb_c_invrs_tau_sfc                =  0.1
>  clubb_c_invrs_tau_shear              =  0.02
>  clubb_c_k1           =  0.75
146a156,157
>  clubb_c_k2           =  0.125
>  clubb_c_k8           =  1.25
147a159,160
>  clubb_c_uu_buoy              =  0.3
>  clubb_c_uu_shr               =  0.3
148a162
>  clubb_c_wp3_pr_turb          =  0.4
159a174
>  clubb_l_e3sm_config          = .true.
172d186
<  clubb_l_use_ice_latent               =  .false.
177a192
>  clubb_nu2            =  5.0
181c196
<  clubb_up2_vp2_factor         =  2.0
---
>  clubb_up2_sfc_coef           =  2.0

[vlarson]

I tried this and I don't think the output is very interesting,

Please try diffing the atm.log.... files. Those should contain a print of the clubb flags and clubb_param parameters. Some of those don't appear in atm_in.

[adamrher]

yeah not very interesting.

There are still lots of new clubb namelists that are not being explicitly set by CAM (via namelist_defaults.xml) on Kate's branch. One example is clubb_l_vary_convect_depth. However, I can see in clubb/model_flags.F90 that it is defaulting to false on the clubb side. But there may be some new namelists that aren't being defaulted to the backwards compatible value in model_flags (i.e., a new feature should be false). Just to give you an idea, there are about 3X more clubb namelists in the new code. You can see this by searching for the "readnl" subroutine in clubb_intr.

clean 053

    namelist /clubb_his_nl/ clubb_history, clubb_rad_history
    namelist /clubbpbl_diff_nl/ clubb_cloudtop_cooling, clubb_rainevap_turb, clubb_expldiff, &
                                clubb_do_adv, clubb_timestep,  &
                                clubb_rnevap_effic,clubb_do_icesuper
    namelist /clubb_params_nl/ clubb_c1, clubb_c1b, clubb_c11, clubb_c11b, clubb_c14, clubb_mult_coef, clubb_gamma_coef, &
                               clubb_c_K10, clubb_c_K10h, clubb_beta, clubb_C2rt, clubb_C2thl, &
                               clubb_C2rtthl, clubb_C8, clubb_C8b, clubb_C7, clubb_C7b, clubb_Skw_denom_coef, &
                               clubb_c6rt, clubb_c6rtb, clubb_c6rtc, clubb_c6thl, clubb_c6thlb, clubb_c6thlc, &
                               clubb_C4, clubb_c_K9, clubb_nu9, clubb_C_wp2_splat, clubb_wpxp_L_thresh, &
                               clubb_lambda0_stability_coef, clubb_l_lscale_plume_centered, &
                               clubb_l_use_ice_latent, clubb_do_liqsupersat, clubb_do_energyfix,&
                               clubb_lmin_coef,clubb_skw_max_mag, clubb_l_stability_correct_tau_zm, &
                               clubb_gamma_coefb, clubb_up2_vp2_factor, clubb_detliq_rad, clubb_detice_rad, &
                               clubb_detphase_lowtemp, &
                               clubb_l_use_C7_Richardson, clubb_l_use_C11_Richardson, &
                               clubb_l_brunt_vaisala_freq_moist, clubb_l_use_thvm_in_bv_freq, &
                               clubb_l_rcm_supersat_adj, clubb_l_damp_wp3_Skw_squared, &
                               clubb_l_predict_upwp_vpwp, clubb_l_min_wp2_from_corr_wx, &
                               clubb_l_min_xp2_from_corr_wx, clubb_l_upwind_xpyp_ta, clubb_l_vert_avg_closure, &
                               clubb_l_trapezoidal_rule_zt, clubb_l_trapezoidal_rule_zm, &
                               clubb_l_call_pdf_closure_twice, clubb_l_use_cloud_cover, &
                               clubb_l_diag_Lscale_from_tau, clubb_l_damp_wp2_using_em

Kate's PR

    namelist /clubb_his_nl/ clubb_history, clubb_rad_history
    namelist /clubbpbl_diff_nl/ clubb_cloudtop_cooling, clubb_rainevap_turb, clubb_expldiff, &
                                clubb_do_adv, clubb_timestep,  &
                                clubb_rnevap_effic,clubb_do_icesuper
    namelist /clubb_params_nl/ clubb_c1, clubb_c1b, clubb_c11, clubb_c11b, clubb_c14, &
                               clubb_C_wp3_pr_turb, clubb_mult_coef, clubb_gamma_coef, &
                               clubb_c_K10, clubb_c_K10h, clubb_beta, clubb_C2rt, clubb_C2thl, &
                               clubb_C2rtthl, clubb_C8, clubb_C8b, clubb_C7, clubb_C7b, clubb_Skw_denom_coef, &
                               clubb_c6rt, clubb_c6rtb, clubb_c6rtc, clubb_c6thl, clubb_c6thlb, clubb_c6thlc, &
                               clubb_C4, clubb_C_uu_shr, clubb_C_uu_buoy, &
                               clubb_c_K1, clubb_c_K2, clubb_nu2, clubb_c_K8, &
                               clubb_c_K9, clubb_nu9, clubb_C_wp2_splat, clubb_wpxp_L_thresh, &
                               clubb_lambda0_stability_coef, clubb_l_lscale_plume_centered, &
                               clubb_do_liqsupersat, clubb_do_energyfix,&
                               clubb_lmin_coef,clubb_skw_max_mag, clubb_l_stability_correct_tau_zm, &
                               clubb_gamma_coefb, clubb_up2_sfc_coef, clubb_detliq_rad, clubb_detice_rad, &
                               clubb_detphase_lowtemp, &
                               clubb_C_invrs_tau_bkgnd, clubb_C_invrs_tau_sfc, clubb_C_invrs_tau_shear, &
                               clubb_C_invrs_tau_N2, clubb_C_invrs_tau_N2_wp2, clubb_C_invrs_tau_N2_xp2, &
                               clubb_C_invrs_tau_N2_wpxp, clubb_C_invrs_tau_N2_clear_wp3, &
                               clubb_iiPDF_type, clubb_ipdf_call_placement, &
                               clubb_l_use_precip_frac, clubb_l_predict_upwp_vpwp, &
                               clubb_l_min_wp2_from_corr_wx, clubb_l_min_xp2_from_corr_wx, &
                               clubb_l_C2_cloud_frac, clubb_l_diffuse_rtm_and_thlm, &
                               clubb_l_stability_correct_Kh_N2_zm, clubb_l_calc_thlp2_rad, &
                               clubb_l_upwind_xpyp_ta, clubb_l_upwind_xm_ma, &
                               clubb_l_uv_nudge, clubb_l_rtm_nudge, &
                               clubb_l_tke_aniso, clubb_l_vert_avg_closure, &
                               clubb_l_trapezoidal_rule_zt, clubb_l_trapezoidal_rule_zm, &
                               clubb_l_call_pdf_closure_twice, clubb_l_standard_term_ta, &
                               clubb_l_partial_upwind_wp3, clubb_l_godunov_upwind_wpxp_ta, &
                               clubb_l_godunov_upwind_xpyp_ta, clubb_l_use_cloud_cover, &
                               clubb_l_diagnose_correlations, clubb_l_calc_w_corr, &
                               clubb_l_const_Nc_in_cloud, clubb_l_fix_w_chi_eta_correlations, &
                               clubb_l_stability_correct_tau_zm, clubb_l_damp_wp2_using_em, &
                               clubb_l_do_expldiff_rtm_thlm, clubb_l_Lscale_plume_centered, &
                               clubb_l_diag_Lscale_from_tau, clubb_l_use_C7_Richardson, &
                               clubb_l_use_C11_Richardson, clubb_l_use_shear_Richardson, &
                               clubb_l_brunt_vaisala_freq_moist, clubb_l_use_thvm_in_bv_freq, &
                               clubb_l_rcm_supersat_adj, clubb_l_damp_wp3_Skw_squared, &
                               clubb_l_prescribed_avg_deltaz, clubb_l_lmm_stepping, &
                               clubb_l_e3sm_config, clubb_l_vary_convect_depth, &
                               clubb_l_use_tke_in_wp3_pr_turb_term, clubb_l_use_tke_in_wp2_wp3_K_dfsn, &
                               clubb_l_smooth_Heaviside_tau_wpxp

And again, lots of these aren't being set on the CAM side, yet they are being read in.

I recognize that parsing through this is a lot of work, so I hope I'm not wrong in suggesting this avenue of exploration. But I could be wrong, so don't feel obliged.

[adamrher]

just now seeing Vince's response. Yes, I believe the atm.log should contain all namelists as they are set on the CAM side (I think they are all contained in these two print statements).

[bstephens82]

Yep I'm aware of those lists and have checked them closely.

[bstephens82]

My tests from a clean 031 tag and from the latest version of the Larson group CAM repo failed, well into the run, with NaN's coming from advance_xp2_xpyp. I wasn't running with the debugger so I don't have that output. Seems strange to me that a clean 031 tag and the latest Larson group repo would crash for similar reasons, I will look into it later today if I have some time. My test with the Nov 11 Larson group CAM tag passed, so I'll also post the SWCF from that when I get a sec.

[bstephens82]

After manually fixing a known bug that existed in the earlier CAM code, my runs for the clean 031 tag, Brian's Nov 11 Larson-group CAM tag, and the Larson-group March 10 repos ran to completion for 1-year tests. It looks to me like Brian's Nov 11 tag and the clean 031 tag have very similar SWCF, not b4b but quite similar.

Clean 031 tag:

Brian's Nov 11 Larson-group tag:

I'm in the process of completing another couple of runs from the Larson-group CAM repo from March 10, the same code base that Kate is incorporating into her PR. I originally was going to use the most recent version of the LG CAM repo but thought I should use March 10 since this might carry more info regarding the code Kate is using. The March 10 repo has opposite settings for the a3_coef clipping and the l_min_xp2 and l_enable_relaxed_clipping flags so I'm toggling those to see what they do (i.e. I'm doing two tests with the March 10 repo). These should be done in a few hours and I'll post them here.

[bstephens82]

For documentation purposes I'll add here that I've gone through the commits marked "BIT_CHANGING" since Brian's Nov 11 Larson-group CAM tag, and the only ones that are actually answer-changing are the things we've previously identified: PDF call put into the "post" position, l_enable_relaxed_clipping turned off, and a3_coef clipping removed. Additionally the l_min_*p2_from_corr_wx flags were turned on in CAM a few months ago (these were set to true in CLUBB standalone years ago).

All other commits marked BIT_CHANGING are just changes that affect the netcdf output but not the answers.

[bstephens82]

possible breakthrough

Ok, when I first ran the March 10 Larson-group CAM repo with everything reverted (using clipping on a3_coef, setting l_min_xp2=F and l_enable_relaxed_clipping=T, and setting PDF placement into the "pre" position), I got these results:

Obviously this shows the same strong enhancement of SWCF that I've been trying to track down and understand. After doing some more digging, I realized that there was a difference between this run and the other two runs (the clean 031 tag and Brian's Nov11 CAM tag), and that was the value of the flag l_do_expldiff_rtm_thlm. I was not familiar with this flag, but it was true in the previous runs and false in the latest run. It turns out CAM was setting this flag to true in the other runs, and after talking with @adamrher I suspected this might be what was causing the problem, because he said this flag being true weakens the SWCF effect by several W/m2. So I did another run with the March 10 Larson-group CAM repo, with this flag set to true in model_flags.F90---et viola!

These results are visually indistinguishable from the Nov 11 repo and not very different from the clean 031 tag. I've tracked down what was causing CAM to set this flag to true---which according to @adamrher it really shouldn't be doing---but that problem appears to be fixed in @Katetc's pull request (the basic explanation is that the flag was being set to true in clubb_intr.F90 in a way that was not very transparent---the line was clubb_config_flags%l_do_expldiff_rtm_thlm = do_expldiff, but this is no longer present in Kate's PR or in the current version of the Larson-group CAM repo).

So this suggests I should be able to get Kate's latest code base to agree with the clean 053 tag in a similar way, which I will try tomorrow and update here again. I've checked that the clean 053 tag also sets the expldiff_rtm_thlm flag to true in the same unintended way, and Kate's code doesn't, so I'm betting this explains the differences we were seeing.

[bstephens82]

The commit that removed the line

clubb_config_flags%l_do_expldiff_rtm_thlm = do_expldiff

(and the whole block that this line was a part of) from clubb_intr.F90 was made by Gunther Huebler on Jan 31 2022, and then Kate merged the Larson-group CAM repo into her PR on Mar 17 so that's how that line got lost. I'm not sure whether Gunther realized the l_do_expldiff_rtm_thlm flag was set differently than the other flags when he made the commit, since it looks like he just thought he was improving the configurable flag setup process. But this may be important for users to be aware of.

[bstephens82]

@adamrher what exact code base were you using here when you did this experiement? I tried a default 1 year run with a clean 053 tag (i.e. not Kate's PR), and then another with the exact same code base except setting l_do_expldiff_rtm_thlm=T in model_flags.F90, and I get identical results. Both say the flag is true in the atm.log. So I'm just confused about how to recover the result you got.

[adamrher]

hmmm ... well the run I plotted above (l_do_expldiff_rtm_thlm) is here: #15

You can peruse into SourceMods and see the mods in model_flags.F90

The cntl in the above plot is here: #19

[bstephens82]

@adamrher I haven't been able to reproduce a test where setting the flag to true in model_flags.F90 creates any difference in the outcome. I was intrigued by the possibility that it might be doing what you said, so I did two tests, but with negative results. The first test was with a clean 053 tag, just a 1-year FHIST run with SE core, and then compared that a run identical in every way except l_do_expldiff_rtm_thlm=T in model_flags.F90. I get identical results:

The I did a second test using the branch you mentioned as your control branch, https://github.com/NCAR/amwg_dev/tree/cam6_3_035.tphysac ---same thing. I did an FHIST 1-year run straight out of the box, then another one where I set l_do_expldiff_rtm_thlm=T in model_flags.in. Again I don't get any difference in the results:

Anyway just FYI I haven't been able to convince myself the model is behaving strangely in respect of this flag, although it may be a moot point with Kate's changes.

[adamrher]

Alrighty. I may be missing something here. And I'll need to look closer into the two runs we did before that showed a 5 W/m2 difference. But I agree this is not the highest priority, since going forward we are explicitly setting l_do_expldiff_rtm_thlm in namelist_defaults.xml.

Can we at least confirm the most important aspect: that we should be defaulting l_do_expldiff_rtm_thlm=F going forward? That is, we don't want to apply down-gradient diffusion of rtm/thlm since that defeats the purpose of computing prognostic fluxes of these quantities? @vlarson ?

[vlarson]

For CAM7, I personally would like to see l_do_expldiff_rtm_thlm=F, because setting it true defeats the purpose of computing higher-order moments.

[bstephens82]

Ok, so when the l_do_expldiff_rtm_thlm flags match (=T), there is basically consistency between the clean 053 tag and Kate's PR:

Clean 053 tag:

Kate's PR code:

There are slight differences but obviously much closer agreement than before. Is this close enough agreement to feel confident that the new CLUBB external is backward-compatible when the settings are reverted?

[vlarson]

If all the flags are reverted correctly, the differences still seem too large to me. They are much greater than round-off. Is the a3_coef clipping, l_relaxed flag, etc., set consistently?

[bstephens82]

Ok I went back through everything and somehow the PDF call placement was back in the "post" position for this run, while it's in the "pre" position for the clean 053 run. I will revert that and see what I get. Everything else I've checked is the same between the two runs.

[bstephens82]

So, latest results, even with the PDF call placement in the "pre" position as it is in the clean 053 tag, this makes only a very slight difference to the SWCF results using Kate's PR code base. Since I was able to demonstrate to my satisfaction that CLUBB has not changed since Brian's November tag as long as you revert a handful of changes, which have been reverted in these tests, my thinking is that Kate's PR may be incorporating some changes to CAM that are causing differences. I will pursue this further tomorrow.

[bstephens82]

I made some more progress this weekend on getting Kate's new PR code to agree with the previous CAM 053 tag. I wasn't having any more luck hunting down different flags or parameters, or answer-changing differences in clubb_intr.F90 (@Katetc suggested a few to check), so I bit the bullet and went back two years in the Larson group CLUBB repo and looked for answer-changing commits. I found a number of commits that will affect answers due to structural changes to CLUBB's code (i.e. not changes to tunable parameter values or configurable flags). Some are very easy to revert (and they help to make new CAM look more like old CAM), but some would take some time to revert since they involve many lines of code.

The list of changes and links to associated commits is here:

1. Brian’s “horrendously BIT_CHANGING” E3SM merge from October 2020: https://github.com/larson-group/clubb/commit/cf5b2e4b80d8bd3ca411b9b5e6d5a2afece04795. However, the only BIT_CHANGING aspect of this commit for the old code, when l_diag_Lscale=F, is that it reduced z_displace from 20 to 10 in ACC, which is easily revertible. This commit has more serious implications if using the taus code (i.e. l_diag_Lscale_from_tau=T), which was inactive in CAM6.
2. Changing the coefficients on the upwp_forcing and vpwp_forcing terms (easily revertible, but should only matter if using prognostic momentum fluxes, i.e. l_predict_upwp_vpwp=T which CAM6 doesn't): https://github.com/larson-group/clubb/commit/0520c4ffabad3f57f549b0c9f24c4e7ef09f7efb
3. Order-of-operations changes when splitting C5 into C_uu_shr and C_uu_buoy: https://github.com/larson-group/clubb/commit/3f75a27dd94673a2b8603c24b989da106e759b39 and https://github.com/larson-group/clubb/commit/bea4ae0588cdc6758927c5c55cd7edcfb8c48c12
4. Changing a threshold for sigma (easily revertible): https://github.com/larson-group/clubb/commit/e9bf3c5582d4a1b079ef35633294ac8b671fcee1
5. wp3 clipping is updated in clip_explicit.F90 (easily revertible): https://github.com/larson-group/clubb/commit/5e53e0cc0bcd487e20e6e22e203defab1bb41a13
6. Removing clipping on a3_coef in advance_clubb_core.F90. https://github.com/larson-group/clubb/commit/4d88fc65090fc186ee942d5247fbb43ddfa67a82
7. Setting l_enable_relaxed_clipping=F in advance_xm_wpxp_module.F90 (this is a hard-coded, non-namelist flag). https://github.com/larson-group/clubb/commit/06d48b1c9da2a8a65f080fa16450d807b97a19ea
8. Replacing division by tau to multiplication by inverse_tau. This would take a lot of time to revert, but one test might be to set all taus and invrs_taus artificially to one, which will wreck physical results but might still provide a way of testing agreement. https://github.com/larson-group/clubb/commit/24b0e2ad62c03197029fff969a57e4ea7abf9a0d
9. Order-of-operations change when splitting C4 and C14. This one is tricky to revert because it took place in I think three different commits, and between those commits, number (8) below happened, which was also answer-changing. The first C4-C14 commit is here, https://github.com/larson-group/clubb/commit/ef3be1b1378bcd1a447f3866c9f88f8cc4764233, a minor related second commit is here, https://github.com/larson-group/clubb/commit/3d3b367cfa0230c0323c2063223aec93a8b4ae58, and there was a final related commit here: https://github.com/larson-group/clubb/commit/edb7bc84684712739876ec803541ba3b165f4a5a. In conjunction with setting all taus and invrs_taus = 1, agreement might be more easily testable.
10. Shixuan Zhang's upwind scheme changes. This one is largely controlled by a new upwind flag, but is answer-changing regardless because it introduced factors of rho and 1/rho that are not wrapped in the new flag: https://github.com/larson-group/clubb/commit/95d7773f81117daff784b8ab91f4211c2edc8390. These changes would take time to revert.

I had been doing one-year runs to test various reversions, but Julio suggested that longer runs might help to smooth out differences from minor order-of-operations changes, etc. When I revert z_displace from (1), the upwp_forcing and vpwp_forcing coefficients from (2), the order-of-operations changes from (3), the sigma threshold from (4), and the magic number for clipping from (5), and run the original 053 tag and the new code for 5 years with all tunable parameters and configurable flags agreeing, I get the following results which are pretty close.

Clean 053 tag:

Kate's PR code (reverted as described):

As you can see now there's only a 0.07 difference in the mean SWCF, with the new code having a slightly stronger effect. I'm thinking that if I were to revert the (6), (7), and (8), we might get back to rounding error only, but I'm not sure if it's worth the work to do it (although my brain thinks it would be fun to try).

[vlarson]

As you can see now there's only a 0.07 difference in the mean SWCF, with the new code having a slightly stronger effect. I'm thinking that if I were to revert the (6), (7), and (8), we might get back to rounding error only, but I'm not sure if it's worth the work to do it (although my brain thinks it would be fun to try).

Nice job! The results look very similar now. I don't think it's worthwhile to revert the other changes.

[bstephens82]

@vlarson and @JulioTBacmeister and I have decided these results are close enough to the original 053 tag that the question of reproducing the same climate has been satisfactorily answered for now. This discussion does contain potentially useful info about reproducing the older CAM version climate however so this discussion may be better locked rather than deleted.

[bstephens82]

For future reference, here are diff output files from the top level directory of CAM and from the CLUBB directory (git diff works for the top level directory but not for the external, hence separate files), showing more clearly what was changed to (mostly) recover the previous results. There is nothing relevant in the associated user_nl_cam file since I did not specify any flags or parameters there.
top_cam_directory.txt
clubb_directory.txt