Elation: Improvements for Find Appointments AI Actions

[belmai]

This merge request includes improvements for the following issues:

• TP-2869: AI Action Improvement: Find Appointments by prompt
• TP-2868: AI Action Improvement: Find future appointment

Key changes:

The LLM component has been extracted to serve both actions. This enables:

• tuning a single prompt,
• easier maintenance,
• consistent behavior between the two actions, and
• reuse of the LLM component for future similar AI actions.

LLM part now has 100% correctness (when checking whether all found appointment match expected appointments):

• LangSmith Experiments.

Callouts:

• AI actions have been renamed after a discussion with Sanne to ensure consistency.
• Explanations are now wrapped in HTML format to support new lines, bullet points, and other formatting elements.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Possible Redundancy The markdownToHtml function is used to convert explanations to HTML format. Ensure that this conversion is necessary and consistent across all outputs, as it adds processing overhead. import { markdownToHtml } from '../../../../src/utils' export const findAppointmentsWithAI: Action< typeof fields, typeof settings, keyof typeof dataPoints > = { key: 'findAppointmentsWithAI', category: Category.EHR_INTEGRATIONS, title: '🪄 Find Appointments with AI (Beta)', description: 'Find all appointments for a patient using natural language.', fields, previewable: false, dataPoints, onEvent: async ({ payload, onComplete, onError, helpers }): Promise<void> => { const { prompt, patientId } = FieldsValidationSchema.parse(payload.fields) const api = makeAPIClient(payload.settings) // First fetch all appointments for the patient const appointments = await api.findAppointments({ patient: patientId, }) // Early return if no appointments found if (isNil(appointments) || appointments.length === 0) { await onComplete({ data_points: { explanation: 'No appointments found', appointments: JSON.stringify([]), appointmentCountsByStatus: JSON.stringify({}), }, }) return } try { // Initialize OpenAI model for natural language processing const { model, metadata, callbacks } = await createOpenAIModel({ settings: {}, // we use built-in API key for OpenAI helpers, payload, }) // Use LLM to find appointments matching the user's natural language prompt const { appointmentIds, explanation } = await findAppointmentsWithLLM({ model, appointments, prompt, metadata, callbacks, }) const htmlExplanation = await markdownToHtml(explanation) // Filter appointments based on LLM's selection const selectedAppointments = appointments.filter((appointment) => appointmentIds.includes(appointment.id), ) const appointmentCountsByStatus = getAppointmentCountsByStatus(selectedAppointments) await onComplete({ data_points: { appointments: JSON.stringify(selectedAppointments), explanation: htmlExplanation, appointmentCountsByStatus: JSON.stringify(appointmentCountsByStatus), Edge Case Handling The logic for handling no matching appointments relies on returning an empty array and a specific explanation. Validate that this behavior is consistent with user expectations and does not cause downstream issues. if (appointmentIds.length === 0) { await onComplete({ data_points: { appointment: undefined, appointmentExists: 'false', explanation: htmlExplanation, }, events: [ addActivityEventLog({ message: `Number of future scheduled or confirmed appointments: ${appointments.length}\n Appointments data: ${JSON.stringify(appointments, null, 2)}\n Found appointment: none\n Explanation: ${htmlExplanation}`, }), ], }) return } // 5. If appointments were found by LLM, return the first matching appointment const matchedAppointmentId = appointmentIds[0] const foundAppointment = appointments.find( (appointment) => appointment.id === matchedAppointmentId ) await onComplete({ data_points: { appointment: !isNil(matchedAppointmentId) ? JSON.stringify(foundAppointment) : undefined, explanation: htmlExplanation, appointmentExists: !isNil(matchedAppointmentId) ? 'true' : 'false', }, events: [ addActivityEventLog({ message: `Number of future scheduled or confirmed appointments: ${appointments.length}\n Appointments data: ${JSON.stringify(appointments, null, 2)}\n Found appointment: ${isNil(foundAppointment) ? 'none' : foundAppointment?.id}\n Explanation: ${htmlExplanation}`, }), ], }) }, } Prompt Complexity The system prompt includes detailed instructions and constraints. Ensure that the complexity of the prompt does not negatively impact the LLM's performance or lead to unintended interpretations. import { ChatPromptTemplate } from '@langchain/core/prompts' export const systemPrompt = ChatPromptTemplate.fromTemplate(` You are a **thorough and precise** medical assistant. You will receive: 1. A list (array) of **appointments** for a single patient (in JSON format). 2. A set of **instructions** (written by a clinician, for a clinician) specifying which types of appointments to find. --- ### **Your Task** Your goal is to carefully analyze the provided instructions and the list of appointments to **identify only those that match the instructions**. - You must be **thorough** in your search but only return results when you are **certain** that they match. - If multiple appointments match, return **all** their IDs. - If no appointments match, return an empty array—this is a valid outcome. - Be **meticulous when evaluating time-based criteria**, ensuring all matches are **precise** relative to \${currentDate}. --- ### **Important Instructions** - **Interpret clinically**: Instructions are written by a clinician for a clinician. Understand medical terminology correctly. - Example: "Rx" relates to prescription or medication follow-up, "PT" refers to physical therapy, "f/u" means follow-up. 2x/wk means 2 times a week while 2:1 means two patients per one clinician. - **Only return matches when you are certain**: - Use your expertise for matching but do not assume connections—appointments must explicitly fit the given instructions. - If an instruction is vague or ambiguous, only match appointments if the data supports it directly. - **Evaluate time constraints precisely**: - If the instruction mentions "past" appointments, **only include** those scheduled **before** \${currentDate}. - If "future" appointments are requested, **only include** those scheduled **after** \${currentDate}. - If an exact date is specified, match it **precisely**. - **Only include appointment IDs that exist in the provided input array**. - **Do not fabricate matches**—returning an empty array is preferable to an incorrect match. ---------- Patient Appointments: \${appointments} ---------- Instructions: \${prompt} Your output must be a valid JSON object with the following fields: - appointmentIds: An array containing the IDs of appointments that match the given criteria. If no appointments meet the criteria, this array must be empty. - explanation: A concise, human-readable explanation that clearly describes: - How and why the selected appointments match the input criteria. - Why any appointments were excluded, if relevant. - Any edge cases or specific reasoning behind your selection. ## Critical Requirements ### Ensure that the explanation strictly matches the appointmentIds: - If appointmentIds contains **selected appointments**, the explanation must accurately describe **why they were chosen**. - If appointmentIds is **empty**, the explanation must clearly state **why no matches were found**. - **Do not contradict yourself**: The explanation must always align with the selected appointments. ### Be thorough and precise: - Carefully **analyze all appointments** before making a decision. - **Double-check** your reasoning to confirm that **only valid matches** are included. - Ensure that no appointments that **do not meet the criteria** are selected. - **Maintain strict consistency** between appointmentIds and explanation.`)

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Handle null API responses gracefully Ensure that the getFutureAppointments function handles cases where the API response is null or undefined to prevent runtime errors. extensions/elation/actions/findFutureAppointmentWithAI/findFutureAppointmentWithAI.ts [30-32] -const appointments = await getFutureAppointments( +const appointments = (await getFutureAppointments( payload.settings as SettingsType, patientId, -) +)) || []; Suggestion importance[1-10]: 9 __ Why: Ensuring the getFutureAppointments function handles null or undefined responses prevents potential runtime errors, which is crucial for maintaining the stability of the application. This suggestion addresses a critical issue effectively.	High
	Add error handling for markdown conversion Add error handling for the markdownToHtml function to ensure that any issues during the conversion process do not cause the entire operation to fail. extensions/elation/actions/findAppointmentsWithAI/findAppointmentsWithAI.ts [63] -const htmlExplanation = await markdownToHtml(explanation) +let htmlExplanation; +try { + htmlExplanation = await markdownToHtml(explanation); +} catch (error) { + htmlExplanation = 'Error converting explanation to HTML'; +} Suggestion importance[1-10]: 8 __ Why: Adding error handling for the markdownToHtml function is a valuable improvement as it ensures the application remains robust and does not fail entirely due to issues in markdown conversion. This change directly enhances the reliability of the code.	Medium
	Validate JSON formatting of appointments Ensure that the formattedAppointments variable is validated to confirm it is a valid JSON string before passing it to the chain.invoke method to prevent runtime errors. extensions/elation/lib/findAppointmentsWithLLM/findAppointmentsWithLLM.ts [25] -const formattedAppointments = JSON.stringify(appointments) +const formattedAppointments = JSON.stringify(appointments); +if (!formattedAppointments) { + throw new Error("Failed to format appointments into JSON."); +} Suggestion importance[1-10]: 8 __ Why: Adding validation for the formattedAppointments variable ensures that the JSON stringification process is successful and prevents potential runtime errors when passing invalid data to chain.invoke. This is a significant improvement for robustness and error handling.	Medium
	Validate arrays before comparison Add validation to ensure that the generatedAppointmentIds and expectedAppointmentIds arrays are not null or undefined before performing comparisons. extensions/elation/lib/findAppointmentsWithLLM/FindAppointmentsWithLLM.evaluate.ts [87-94] const isEqual = Array.isArray(generatedAppointmentIds) && Array.isArray(expectedAppointmentIds) && + generatedAppointmentIds !== null && + expectedAppointmentIds !== null && generatedAppointmentIds.length === expectedAppointmentIds.length && generatedAppointmentIds.every((id, index) => id === expectedAppointmentIds[index]) Suggestion importance[1-10]: 6 __ Why: Adding validation to ensure arrays are not null or undefined before comparison is a minor but useful improvement. It enhances code safety and prevents potential errors during runtime.	Low
Security	Replace sensitive URLs with placeholders Replace hardcoded URLs in the mock data with placeholders to avoid exposing sensitive or environment-specific information. extensions/elation/actions/findAppointmentsWithAI/testdata/GetAppointments.mock.ts [24-25] -anonymous_url: 'https://sandbox.elationemr.com/appointments/141701667029082/patient-forms/?key=642301d3930ac1e4d052ff65c093c5f1da1697e6b861a18f43a042b5afca50a1', +anonymous_url: 'https://sandbox.elationemr.com/appointments/{appointment_id}/patient-forms/?key={key}', Suggestion importance[1-10]: 7 __ Why: Replacing hardcoded sensitive URLs with placeholders is a good practice to avoid exposing sensitive or environment-specific information in the codebase. This change improves security and maintainability.	Medium
	Sanitize inputs for safety Add a mechanism to sanitize or validate the prompt and appointments inputs to ensure they do not contain malicious or malformed data before being used in the systemPrompt. extensions/elation/lib/findAppointmentsWithLLM/prompt.ts [35-38] Patient Appointments: -\${appointments} +\${sanitizeInput(appointments)} ---------- Instructions: -\${prompt} +\${sanitizeInput(prompt)} Suggestion importance[1-10]: 7 __ Why: Sanitizing or validating the prompt and appointments inputs enhances security by preventing potential injection attacks or malformed data issues. While not critical in this context, it is a good practice to ensure input safety.	Medium