REINFORCEMENT LEARNING FOR DYNAMIC TREATMENT OF CHRONIC KIDNEY DISEASE MINERAL AND BONE DISORDER (CKD-MBD) AMONG PATIENTS WITH KIDNEY FAILURE ON HEMODIALYSIS: A DEMONSTRATION PROJECT

Treatment of chronic kidney disease mineral and bone disorder (CKD-MBD) involves dynamic titration of multiple drug classes including vitamin D agonists (vit D), calcimimetics (calm), and phosphorus binders. Developing dynamic treatment algorithms that guide titration could facilitate needed randomized controlled trials (RCTs) of alternative CKD-MBD treatment strategies. 

In this demonstration project, we used monthly electronic health record data from adult patients receiving in-center hemodialysis (HD) at one of 12 units at our institution between 2006-2024 and including any months after first parathyroid hormone (PTH) ≥400 pg/ml. CKD-MBD medication doses were harmonized to common daily equivalents. Months between 2006-2021 were split into 60% training and 40% validation at the patient level. All data from 2022-2024 were included in a held-out test set. Months with medication titrations were up-sampled 3 times in the training set to increase informative examples. Behavioral cloning (BC) and reinforcement learning (RL) were fit using a recurrent neural network to construct different treatment policies involving a matrix of increase/initiate, maintain, and decrease/stop for each of vit D and calm, yielding 9 potential actions. Covariates included CKD-MBD laboratories (albumin-corrected calcium, serum phosphorus, PTH), phosphorus binders (total and calcium-containing), baseline doses of vit D and calm, demographics, dialysis vintage, calendar year, and diabetes, each over the prior 3 months. RL was initialized with BC and rewarded future CKD-MBD laboratories within range (i.e., PTH 400-600 pg/ml; calcium 8.4-10.0 mg/dl; phosphorus 3.5-5.0 mg/dl) or moving toward range (PTH) and conferred penalties for moving away from range (PTH) or out of range (calcium, phosphate), each with discounting over time. A weighted solution including 90% RL and 10% BC in the loss function was selected to prevent extreme deviations from practice, based on interim clinical review during model fitting. An ensemble of 25 models with distinct starting seeds was used to improve stability.

Our weighted RL policy recommended fewer months in which both drug classes were maintained (51%) and recommended more titration of calm (12% of months) than actual practice, referred to as ground truth (GT; maintained medications in 71% of months; titrated calm in 5% of months; both p<0.001). The odds of RL recommending a net increasing (i.e., increase vit D, calm, or both), net decreasing (i.e., decrease vit D, calm, or both), or no net action (i.e., maintain both medications or one increased and one decreased) compared to GT across different CKD-MBD starting states in the validation dataset is depicted in the Figure (n=1,047 patients with 36,799 observed months). CKD-MBD states are defined by corrected-calcium [1: <8.0; 2: 8.0-8.4; 3:8.4-10.0; 4:>10.0 mg/dl], serum phosphorus [1: <3.5; 2: 3.5-5.0; 3: 5.0-6.5; 4: 6.5-8.0; 5: >8.0 mg/dl] and PTH [1: <150; 2: 150-400; 3: 400-600; 4: 600-900; 5: >900 pg/ml]. The RL policy was evaluated in a held-out test set using a nearest neighbor matching approach to pair observations in which the GT actions taken were concordant versus discordant with RL policy (n=1,109 patients with 20,591 observations prior to matching). For actions concordant with RL policies, the odds of being out of range in the next month were reduced by 35% for PTH and 16% for calcium (both p<0.001). 

We conclude that preliminary weighted RL-derived policies show promise for generating dynamic treatment algorithms to test in RCTs. This abstract was previously presented, in part, at the Gordon Research Conference on Physiology, Biology and Pathology of Phosphate in February 2025 and the Renal Research Institute Symposium in June 2025.