Welcome. This session takes about sixty minutes live, or about the same on your own with the pauses. Here is the plan. First, four words, AI, machine learning, deep learning, and generative AI, and how they fit inside each other. Then one model from start to finish: the eGFR on a chemistry panel, where its number comes from, how a model is fit to data, and what happens when you add a predictor. Then the fitting loop that every model comes from, and two questions that sort any model. After that, what goes wrong. A model can be too flexible. A neural network is the same loop with far more dials. And a score on cases a model has already seen proves nothing. Then back to the clinic, and what this changes on the wards. The questions along the way are live: you answer on your phone or laptop. Thursday starts with the chatbot, which is the same idea with words. LIVE: Run sheet, with minutes per block, for about 60 minutes of talk and 3 for questions. Before class: open this deck once with ?presenter=YOUR-KEY so the live questions open. 0 to 9 (9 min): slides 1 to 5; slide 4 is the setup (target, model, predictors, error, loss), about a minute and a half, with one click for the loss box; slide 5 is height and weight, move the two sliders yourself for about two minutes, press “Show the best fit”, click once to name the fitting loop, then click again for “What is missing?” and take two or three answers aloud. 9 to 19 (10 min): slides 6 to 11, with the live question on slide 8 (about 45 seconds, then press A); on slide 10 the room has already fit a line, so move the sliders for only about a minute, then press “Show the best fit”; on slide 11 ask the room which way the average miss will move, then turn on “Add sex”. 19 to 35 (16 min): slides 12 to 20; slide 14 one minute, pointing at the padlock; slide 15 two minutes; slides 18 and 19 are live questions too, about 45 seconds each, then press A to reveal; slide 20 ninety seconds, pointing at the Type column and then at the last column. 35 to 52 (17 min): slides 21 to 27; on slide 21 move the slider yourself, about two minutes, and let the room call out where it goes wrong; slide 22 two minutes; slide 25 three minutes, as a callback to Tuesday’s talk; slide 27 is the bridge to Thursday, one minute. Keep about two minutes in this block for questions from the room. 52 to 60 (8 min): slides 28 to 31; read slide 30 in one minute; end on the Thursday line on slide 31. 60 to 63: questions, on slide 35, the last slide (press End from slide 31). Slides 32 and 33 are for students afterwards; slide 33 has the TensorFlow Playground exercise. Slide 34 is the reference list. LIVE: Turn on Zoom live captions and paste the deck link into the chat at the start. Paragraphs in these notes that start with PAUSE are for the recorded version; skip them live. ChatGPT Edu and Microsoft Copilot are this course’s institutional tools; swap in your school’s. Click outside a slider or button before using the arrow keys.

Yesterday’s talk was the history: how we got from rules written by hand to the chatbots of today. Here are four words from that history, and how they fit together. If you missed yesterday, nothing today depends on it. Artificial intelligence is the largest circle. The sizes of the circles aren’t meant to show how big each field is. It means a computer doing a task that seems to need judgment, by any method. MYCIN, from the 1970s, advised on antibiotics with rules that experts wrote by hand. Machine learning is inside it. The rules aren’t written by hand. They are learned from data. The eGFR equation on every chemistry panel is an example. A statistician would call it a regression. The line between statistics and machine learning is blurry, and for this course a model learned from data counts. Deep learning is inside machine learning. It uses a neural network: many layers of simple units, each passing numbers to the next. Models that read images, such as chest X-rays, are usually built this way. Generative AI is inside deep learning. These models write new text or make new images. A chatbot is the example you know. So a chatbot is generative AI, and it’s also deep learning, machine learning, and AI. The words overlap, but each one means something specific. Today we start in the second circle, with the eGFR, and reach the third, deep learning. Thursday we reach the smallest circle, the chatbot. The same idea, learning from data, is under all of them.

Every model in this talk has the same setup, and it’s worth one slide. Person number i has a target, which is the thing we want to predict, and a set of predictors, which are the things we know about them. The model is a function: it takes the predictors and returns a prediction. The prediction is almost never exactly the target. What’s left over is the error. So the target equals the model’s prediction plus the error. The target can be a number, like weight, or the chance that a patient falls in the hospital. It can also be a category: benign or malignant, pneumonia or not. The error isn’t only that each person is a little different. It also holds any error in measuring the target, and in measuring the predictors. My PhD is in human genetics, and one of our constant problems was defining the disease. If you can’t define it well, it’s very hard to find the gene that goes with it. The best models start with the best measurements. Those four words, target, model, predictors, error, will come back all session. The eGFR is a model. A chatbot is a model. The f can be as simple as a straight line or as complex as a neural network. \[Click\] How do we choose the model? We need a way to score it. A loss function adds up the errors across everyone into one number. One common choice squares each error and adds them up, which is called the residual sum of squares. Another is the plain average miss. Big errors make a big loss. Fitting a model means choosing the version of the function that makes the loss as small as it can be. On the next slide, we do that by hand.

Made-up data for 40 adults.

``` {ojs} { const W = 640, H = 400, l = 64, r = 14, t = 12, b = 54, x0 = 145, x1 = 205, y0 = 30, y1 = 140; const X = h => l + (h - x0) / (x1 - x0) * (W - l - r), Y = w => H - b - (Math.min(Math.max(w, y0), y1) - y0) / (y1 - y0) * (H - b - t); const f = hwLine(hwA, hwB); return html``; } ```

**What is missing?** What else would help predict weight? In other words, what additional predictors should we consider?

weight = a + b × height + error

``` {ojs} viewof hwA = rangeInput({ label: "a", min: -150, max: 50, step: 1, value: -20 }) viewof hwB = rangeInput({ label: "b", min: 0, max: 1.6, step: 0.01, value: 0.4 }) ``` **a**: the intercept. **b**: the slope, kg per extra centimeter. ``` {ojs} { const miss = hwMiss(hwA, hwB), rss = hwRss(hwA, hwB); const best = Math.abs(hwA - hwBest.a) < 0.5 && Math.abs(hwB - hwBest.b) < 0.005; return html`

Loss (sum of squared misses): ${rss.toFixed(0)}Average miss: ${miss.toFixed(1)} kg

${best ? html`

The best fit: no line has a smaller loss.

` : ""}`; } ``` ``` {ojs} { const button = html``; button.onclick = () => { for (const [view, v] of [[viewof hwA, Math.round(hwBest.a)], [viewof hwB, +hwBest.b.toFixed(2)]]) { const input = view.querySelector("input"); input.value = v; input.dispatchEvent(new Event("input", { bubbles: true })); } }; return button; } ```

What you just did is the **fitting loop**: guess the dials, score the misses with the loss, adjust, repeat.

Here is that setup with the simplest model there is. The target is weight. The one predictor is height. Each dot is one made-up adult. The model is a straight line with two numbers in it: a, the intercept, and b, the slope. Each thin line is that person’s error, how far their weight is from the line. The loss is the sum of the squared errors, and the average miss is beside it. PAUSE: 2 min. Move the two sliders and make the loss as small as you can. Then press “Show the best fit”. I could move these sliders all day long, but it turns out we have good ways of fitting a model. Fitting is what the computer does when you press that button: it finds the line with the smallest sum of squared errors. For a straight line, that’s the best you can do. This is called linear regression, and it’s machine learning in its simplest form. \[Click\] What you just did by hand has a name: the fitting loop. Guess the dials, score the misses with the loss, adjust the dials, and repeat. When to stop is a question we come back to. For a straight line there’s a formula that jumps to where the loop would end, which is what the button uses. Bigger models, like neural networks, really do run the loop, lap after lap. \[Click\] Even the best line leaves errors. Some tall people are light and some short people are heavy. So what’s missing? What else would help us predict weight? Age, sex, how much of the weight is muscle, waist size. Each one is another predictor, another column in X. The eGFR does this with creatinine, age, and sex. LIVE: About two and a half minutes. Move the sliders yourself and ask the room which way to move b. Press the button. Click once to name the fitting loop, then again for the last question, and take two or three answers aloud.

Nobody measured her glomerular filtration rate. \[Click\] Measuring it means infusing a marker that the kidney filters and does nothing else with, such as iohexol, and timing how fast it clears. \[Click\] That takes hours, and it’s rarely done in routine care. \[Click\] The fifty-seven was computed from three things the lab did have: her creatinine, her age, and her sex. Something turned those three numbers into a kidney function, and you trusted it enough to clear her for contrast. \[Click\] So where did that equation come from? Not from the physiology of nephrons. It was learned from data. A few thousand people did have their GFR measured. Their creatinine, age, and sex were given to a computer with one instruction: find the formula that best predicts the measured number. \[Click\] The eGFR is a model. It’s the model you already trust most, and you have probably never asked what it was trained on. Remember this patient. She weighs forty-five kilograms and has lost a good deal of muscle over a year of illness. We’ll come back to her at the end. Nobody has told her that a model had a part in the fifty-seven.

Illustrative data, not the CKD-EPI data. A simpler curve than the real eGFR.

``` {ojs} { const W = 640, H = 400, l = 64, r = 14, t = 12, b = 54, ymax = 170; const X = cr => l + (cr - 0.5) / 3.5 * (W - l - r), Y = g => H - b - Math.min(g, ymax) / ymax * (H - b - t); const f = dialCurve(dialA, dialB); const grid = Array.from({ length: 141 }, (_, i) => 0.5 + i * 0.025); const path = "M" + grid.map(c => `${X(c).toFixed(1)},${Y(f(c)).toFixed(1)}`).join("L"); return html``; } ```

GFR = a × creatinine^−b

``` {ojs} viewof dialA = rangeInput({ label: "a", min: 30, max: 150, step: 1, value: 50 }) viewof dialB = rangeInput({ label: "b", min: 0, max: 2, step: 0.05, value: 0.3 }) ``` **a**: the GFR when creatinine is 1.0. **b**: how fast GFR falls as creatinine rises. ``` {ojs} { const miss = dialMiss(dialA, dialB), best = dialA === dialBest.a && dialB === dialBest.b; return html`

Average miss: ${miss.toFixed(1)} mL/min/1.73 m²

${best ? html`

The best fit: no setting of a and b misses by less.

` : ""}`; } ``` ``` {ojs} { const button = html``; button.onclick = () => { for (const [view, v] of [[viewof dialA, dialBest.a], [viewof dialB, dialBest.b]]) { const input = view.querySelector("input"); input.value = v; input.dispatchEvent(new Event("input", { bubbles: true })); } }; return button; } ```

**Fitting** means a computer turns these dials until the error is as small as it can get.

Here is fitting, done by hand. Each dot is one made-up person: creatinine across, measured GFR up. The thick curve is a simple equation with two numbers in it, a and b. Those two numbers are the dials. The number a sets how high the curve sits: it’s the GFR the curve gives at a creatinine of one. The number b sets how fast GFR falls as creatinine rises. Each thin line is a miss: how far one person’s measured GFR is from the curve. The number beside the sliders is the average miss. PAUSE: 1 min. Move the two sliders and make the average miss as small as you can. Then press “Show the best fit”. \[Click\] Fitting means a computer turns these dials until the error is as small as it can get. This is the same fitting loop you ran with height and weight. Here the computer tries every setting the sliders allow and keeps the best one. Real fitting uses faster methods, with the same goal. The real eGFR equation has a few more numbers in it, and it was fit to eight thousand two hundred fifty-four people whose GFR was measured, not to forty made-up dots. A chatbot is the same idea with billions of numbers. The data on this slide are made up, and the curve is simpler than the real equation. Its numbers aren’t the real eGFR’s. LIVE: About one minute; the room fit a line on slide 5. Move the sliders briefly, press the button, then click for the last line.

Illustrative data, made up. Women are built to have a lower GFR at the same creatinine.

``` {ojs} { const W = 640, H = 400, l = 64, r = 14, t = 12, b = 54, ymax = 170; const X = cr => l + (cr - 0.5) / 3.5 * (W - l - r), Y = g => H - b - Math.min(g, ymax) / ymax * (H - b - t); const w = addFit(useSex); const grid = Array.from({ length: 141 }, (_, i) => 0.5 + i * 0.025); const curve = fem => "M" + grid.map(c => `${X(c).toFixed(1)},${Y(addPredict(w, { cr: c, fem }, useSex)).toFixed(1)}`).join("L"); const dot = p => useSex ? (p.fem ? ink.terracotta : ink.teal) : ink.ink; return html``; } ```

``` {ojs} toggleInput = label => { const box = html` ${label}`; const input = box.querySelector("input"); box.value = false; input.onchange = () => { box.value = input.checked; box.dispatchEvent(new Event("input", { bubbles: true })); }; return box; } viewof useSex = toggleInput("Add sex") ``` ``` {ojs} { const sc = addScore(addFit(useSex), useSex); return html`

Numbers in the model: ${useSex ? 3 : 2}

Average miss: ${sc.miss.toFixed(1)} mL/min/1.73 m²

Sum of squared misses (RSS): ${sc.rss.toFixed(2)}

`; } ``` Dots: teal men, terracotta women. Thin lines: the miss for each person. The fit makes the RSS as small as it can, on a log scale.

**Fitting** with a second predictor is the same job: find the numbers that make the misses smallest.

Now add a second predictor. The real eGFR equation uses sex as well as creatinine and age, so we do the same. Same made-up people, same idea, but now the equation can have a different level for women and men. PAUSE: 1 min. Turn on “Add sex” and watch the average miss and the RSS. The fit now gives each sex its own curve, and both numbers fall. In this made-up data, women have a lower GFR at the same creatinine, so a model that knows the sex can follow both groups. Statisticians call the sum of squared misses the RSS. The fit makes it as small as it can, so the RSS can only go down when a predictor is added. Whether the drop is real, or only a better fit to these forty people, is a different question, and we come back to it with the slider in Part 2. \[Click\] Fitting with a second predictor is the same job as before: find the numbers that make the misses smallest. The data are made up. The real equation has more terms, and it was fit to eight thousand two hundred fifty-four people whose GFR was measured. LIVE: About one minute. Ask the room which way the average miss will move, then turn it on.

The eGFR story contains almost every word you need for the rest of the course. Predictors are the inputs: creatinine, age, and sex. The response is the thing being predicted: the measured GFR. The training set is the eight thousand two hundred fifty-four adults whose numbers were used to fit the equation. They came from ten studies and were mostly research participants. About a third were Black. The validation set is a second group held back while the model is being fit. It’s used to choose between versions: when to stop, or which of two models to keep. The test set is checked once, at the end. For the eGFR, that was four thousand fifty people from twelve other studies that the equation had never seen. Watch the names, because they vary by community. The response is what statisticians call it; machine learning says target, which is the word we used on the setup slide, or label. Same idea. And the eGFR paper calls those four thousand fifty people its validation data set, and many medical papers use the word that way. In machine learning, a group held back and checked once at the end is the test set. When you read a paper, ask what the held-back cases were used for, not what they are called. And the model is the equation itself, once it has been fit. The letter f stands for function, which just means a rule that turns inputs into an output. Fit means that a computer tried numbers in the equation until its predictions were as close to the measured values as they could get. For the eGFR, the model is a handful of numbers. For a chatbot, it’s billions. The role is the same.

Here are those three sets as a picture. Start with one table, one row per patient. For the eGFR that’s creatinine, age, and sex, the predictors, and the measured GFR, the target. Every row has the answer. Before any fitting, you split the rows. Most go into the training set, and that’s what the fitting loop turns the dials on. A smaller slice is the validation set. The loop gets to look at it, to decide when to stop and which model to keep, so think of it as a pseudo test set. And the last slice is the test set. It’s locked. Nobody looks at it while the model is being built, and you check it once, at the very end. That’s the number you report. The shares on the slide are typical, not a rule. Two details matter more than the percentages. Split by patient, so the same person never lands in two sets. And split before you fit, because once the test cases have influenced a choice, they’re no longer a test. The eGFR researchers did one better. Their test set wasn’t a random slice of the same table. It was four thousand fifty people from twelve other studies. A test on different people, from different studies, is harder to pass, and it tells you more.

Here is the history of one equation, rebuilt from the numbers in the papers. Read the last column as a pattern, not as nephrology. MDRD was fit to one thousand and seventy people who already had kidney disease, and tested on five hundred fifty-eight more. In healthy people it gave values that were too low, and healthy people were labelled as diseased. The 2009 equation was more accurate at higher GFR, and it included a term that adjusted the result for Black race. Even with that term, the 2021 study found it overestimated measured GFR in Black adults. An overestimate makes kidney disease look milder than it is. The 2021 refit removed race, and it’s still a few points off in both directions. A version that adds cystatin C, a second blood marker that doesn’t depend on muscle, is more accurate. \[Click\] Every row says the same thing. The model was fit to these people, and here is what happened to people unlike them. The training population is part of the model, as much as the equation is. That’s a take-home message, and we’ll come back to it. Why the race term was there, and who was harmed by it, is the subject of the bias and equity session later in the course. LIVE: If asked why the race term was added: in the 2009 data, Black participants on average had a higher measured GFR at the same creatinine, and the term adjusted for that. The 2021 paper states that race in these equations is a social and not a biologic construct. In 2021 a national task force recommended the refit without race for all US laboratories, and wider use of cystatin C to confirm the estimate. What the term did to Black patients is the bias and equity chapter’s subject.

Here is the fitting loop as a picture. You ran it by hand with the height and weight sliders. Step one, guess the dials; anywhere will do. Step two, use them to predict every training case. Step three, score the predictions: add up the misses into one number, the loss. Step four, adjust the dials in the direction that lowers the loss. Then go around again. When do you stop? Not when the training loss stops shrinking. Look at the chart. The red line is the loss on the training cases, and it keeps falling: with enough laps, a model can fit the cases it learns from almost perfectly. The gold line is the loss on a second set of cases the loop never fits to, the validation set. It falls, bottoms out, and then rises. From there on, the model is memorizing the training cases, noise included, instead of learning the pattern. That’s overfitting. So the loop stops where the validation loss is lowest. The validation set is also how you choose between models, for example a simple one and a flexible one. You’ll see that happen with the slider in a few minutes. Think of it as a pseudo test set: the loop gets to look at it, so it can’t be the final word. Then you leave the loop, and there’s one more step. Step five: test the model once, on a third set of cases, the test set, that neither the loop nor the choices ever saw. The loss inside the loop only tells you how well the model fits the cases it was built from. The test set tells you how it does on new ones. We’ll see what happens when that step is skipped or done badly. For a straight line, and for the eGFR equation, there’s a formula that lands where the loop would end, so nobody needs to go around. For a neural network there’s no such shortcut, and the computer really does run this loop, many times. The strip at the bottom is the whole path. Fitting needs training cases with both the predictors and the truth. The loop is checked on the validation set as it runs. The model it gives, f, is tested once on the test set. Then a new patient arrives with only the predictors, and f turns them into a prediction. The eGFR, a sepsis alert in the EHR, and a chatbot differ in the data and in how many dials they have. The loop is the same. So there are two take-home messages here. There’s a loop. And there’s such a thing as overfitting: a model can do very well on the cases it was fit to, and poorly on new ones.

Two questions sort any model in about ten seconds. Start at the top. First: does the training data contain the answer? For the eGFR, yes. Someone measured every one of those GFRs. That’s supervised learning. If there’s no answer column, only patterns to find, it’s unsupervised learning. The example is breast cancer subtypes, which were found by grouping tumors that share gene activity, with nobody telling the computer the groups ([Perou et al., 2000](#ref-pmid:10963602)). We won’t say more about unsupervised learning today, but it’s worth knowing it exists. Second, and only for a supervised model: what does it output? A number makes it a regression. The eGFR is a regression, and so is a model that reports an ejection fraction. In the picture, each red line is how far a prediction misses. A category makes it a classification: benign or melanoma, sepsis or no sepsis. Here the model draws a boundary, and a case on the wrong side is an error. An unsupervised tool that sorts cases into groups isn’t a classification, because nobody told it what the groups were. \[Click\] The difference matters for how you judge a model. A regression is wrong by an amount. A classification is wrong by a kind, a miss or a false alarm. A miss costs the patient who was missed. A false alarm costs the staff who answer it, and the patient who is worked up for nothing. Sometimes it isn’t obvious which kind a model is. Sorting that out is a good first step to understanding how it was built and how it’ll be used.

Four fields, the same two questions. Read the Type column, then the last column. In cardiology, a model reads ultrasound video and gives a number, the ejection fraction. In pediatrics, a model reads a hand X-ray and gives a bone age in months. Both are regressions: they are wrong by an amount. In dermatology and pathology, a model reads an image and gives a category. Both are classifications: they are wrong by a kind. Now look at the last column, because the same lesson is in it. The cardiology model was off by four point one points at the hospital where it was built, and by six point zero at another health system. In the pathology challenge, the best entry scored zero point nine nine four on the area under the curve, and the worst scored zero point five five six, barely better than a coin flip. Results on the authors’ own test data are the optimistic number. Ask where it was tested, and by whom. For the record, three more examples are listed under the table: an ECG rhythm classifier, tumor subtypes found without any diagnoses, and the diabetic retinopathy system from Tuesday. They are here so you can find the papers. LIVE: About one minute and a half. Don’t read the table. Point at the Type column and then at the last column.

Move the slider and watch. I simulated these data, so I know exactly where they came from: the dashed line is the real pattern, which the model never gets to see. The filled dots are the cases the model learns from. The hollow dots were held back for testing, and while we fit, we pretend they don’t exist. Under the charts, the two numbers are the error on each set. At the far left the model is too simple. It’s a flat or straight line through a curve, and it’s wrong everywhere, on both sets. Statisticians call that kind of error bias, in a technical sense that has nothing to do with the social meaning. Now let the model bend. As you slide right, the training error goes down, and for a while the test error goes down with it. Then, somewhere in the middle, something happens. The training error keeps falling, but the test error starts to climb. Keep going and it gets kind of absurd: the curve bends through every filled dot, the training error drops to zero, and the test error goes off the chart. The model has learned the random scatter in these twelve cases along with the real pattern. That error is called variance. That’s overfitting, and here’s the real definition, from a standard machine learning textbook. A model overfits when a simpler model would do worse on the training cases but better on new cases from the same population. Two things in that definition matter. First, it compares two models. Test error is almost always a little higher than training error, and that alone isn’t overfitting; the sign is that more flexibility keeps lowering the training error while the error on held-back cases rises. Second, it’s about new cases from the same population. A model that fails at a different hospital, on different patients, has a different problem, and we’ll see it on the sepsis slide. If you were only ever shown the training error, you’d pick the worst model. This is the job of the validation set in the fitting loop: choose the setting where the error on held-back cases is lowest, and keep a separate test set for the end. And if you find you’ve overfit, the fix is to go back and make the model simpler. PAUSE: 2 min. Pause here and move the slider from one end to the other.

Here is a neural network. It’s the fitting loop again, with a far bigger equation. \[Click\] On the left are the inputs. For an image, these can be the raw pixel values, so nobody has to decide in advance what to look for. \[Click\] In the middle are layers of simple units. Each unit adds up what it receives, with its own dials, and passes the result to the next layer. On the right is the output: a number, such as an ejection fraction, or a category, such as melanoma. You don’t have to tell it what to look for, but you do still need the answer for every training case. \[Click\] Every line in this picture is a dial. This toy network has fifty. The networks behind the four examples have far more. GPT-3, an early ancestor of ChatGPT, had a hundred and seventy-five billion ([Brown et al., 2020](#ref-brown2020)). Turning that many dials takes an enormous amount of computing time and energy, which is a big part of why a new chatbot model costs so much to build. Fitting is the loop you already know: turn every dial until the misses on the training cases get smaller. The word deep in deep learning means many layers. Everything from the slider still applies. A network with that many dials can memorize its training cases, so the test set matters even more. And what it learned depends on the people in its training data. LIVE: About one minute. Don’t explain the arithmetic. The picture is for the idea of many dials, and the next section uses it for words.

The slider showed one trade-off. Here is a second one that comes with it. The eGFR equation is a handful of numbers. You can read each one and check it against what you know. Higher creatinine gives a lower eGFR, older age gives a lower eGFR, and you could work it out by hand. A few thousand people were enough to fit it. The cost is that it can only follow the shape its equation allows. A neural network has thousands to billions of numbers. It can fit patterns no short equation can, such as what a pneumothorax looks like on a film, or how a sentence goes on. But nobody can read those numbers one by one, so it’s harder to explain why it gave a particular answer. And it needs far more data, or it fits the noise, as the slider showed at the far right. Turn that around: with only a few hundred patients, you can’t fit a very complex model. That’s one reason the internet and the electronic health record mattered so much in yesterday’s history. Machine learning needs a lot of data. \[Click\] So the more flexible the model, the more patterns it can fit, the less you can explain, and the more data it needs. A model is called interpretable when a person can read why it gave its answer. Yesterday’s expert systems were very interpretable: they could tell you the rules they followed. Neither kind is better in general. For a lab value, a simple model you can check is often the right choice. Images and language need the flexible kind, and then the test set matters even more. LIVE: One minute. Read the last line and move on.

Here is the mistake the eGFR researchers avoided, and the mistake many vendor slides make. Suppose you fit a formula to eight thousand people, and then you report how well it predicts those same eight thousand people. You have measured how well the formula memorized them, not how well it will work on the next patient. A flexible enough model can fit its training data perfectly and still be useless. \[Click\] That’s why a separate test set is the least you should accept. Fit on one set of cases, and report performance on a different set the model never saw. If someone tells you their model’s error is near zero, your first question is: on the training data or the test data? Did you try it on new data, and what was the error there? The same goes for a model trained on synthetic patients. Synthetic training data can help, but the test still has to be on real ones. When someone gives you one number and doesn’t say which set it came from, you do not have a number yet. \[Click\] The usual rule uses three sets, the same three as in the fitting loop. The training set is used to fit the model. The validation set is used to choose between models, for example between a simpler one and a more flexible one. The test set is kept aside and used once, at the end. If the test set is used to choose, it becomes a second validation set, and its number is optimistic too. The next slide shows why this matters.

You saw this model yesterday. Here is why it belongs in today’s talk: it’s the same failure as the slider, in the clinic. One EHR vendor’s sepsis model, the Epic Sepsis Model, was in use at hundreds of US hospitals in 2021. One academic center tested it on thirty-eight thousand four hundred fifty-five of its own hospitalizations. A test on patients from a different setting than the ones the model was built on is called external validation. The area under the curve was zero point six three, where zero point five is no better than chance. Of every hundred patients who had sepsis, the model missed sixty-seven. And since it alerted on eighteen of every hundred admissions, when only about seven had sepsis, most alerts were for patients who didn’t. The grid shows the same counts, scaled to two hundred admissions. Each box is one admission. The colored boxes are the thirty-six where the alert fired. The dots are the thirteen patients who had sepsis. Only four of the dots sit in a colored box. The other nine had sepsis and no alert. The study doesn’t show why. A common reason is that the patients a model meets in use differ from the patients it was built on. The name for that is dataset shift. For the wards: no alert does not mean no sepsis. And if you silence an alert, you should be able to say why. LIVE: About three minutes. They saw this study yesterday. Walk the grid (colored boxes, then dots), then name the link to the test set.

That gives you another of the course’s recurring lessons: the published number is the optimistic one. A model’s performance on the cases it was built from is usually the best it will look. On new patients, especially from another hospital, it’s usually lower. \[Click\] So ask three things of any model. What was it trained on? What was it tested on, and by whom? If the training set and the test set are the same cases, the score tells you nothing yet, because the model has already seen the answers. Why ask what it was trained on? Because the training data are part of the model. A model built on patients in one country, or at one kind of hospital, may not work the same way somewhere else. Often you won’t get the details: companies frequently don’t publish what a commercial model was trained or tested on. Ask anyway. But there’s a second answer that doesn’t depend on them, which is to test it on your own patients. I sit on the committee that vets machine learning models for our health system, and one of our first questions about any model is what it will cost to localize. Localizing means running the model on our own past patients, say the last six months or the last few years, and scoring it against what we know happened. Take a fall-risk model. Of the patients who fell, how many did the alert fire for? And how often did it fire for patients who didn’t fall? Sometimes the next step is to tune the model to fit our patients better. And it doesn’t stop at go-live. Patients change over time, and so do the records; even the codes used for diagnoses have changed their definitions over the years. So someone has to keep monitoring the model to make sure it still works.

Take any sentence from the internet. The patient was started on empiric vancomycin. Now turn it into a table, the same kind of table as the eGFR. The predictors are the words so far. The truth is the very next word. “The” is followed by “patient”. “The patient” is followed by “was”. And so on, until “the patient was started on empiric” is followed by “vancomycin”. You might ask: lots of sentences start with “The”, and many different words can come next. That’s why the model uses all the words so far, not just the last one. The more of the sentence it sees, the better it can guess. \[Click\] Nobody had to label anything, because the next word is already in the text. Every sentence ever written becomes training cases with no labelling work. That’s why these models could be trained on a very large fraction of the written internet. Thursday starts here: a chatbot is machine learning on words, and the same test-set questions apply. LIVE: About one minute. Stop here. This is the bridge to Thursday.

Go back to the clinic. The sixty-two-year-old woman is still on the schedule. \[Click\] She weighs forty-five kilograms and has lost muscle over a year of illness. \[Click\] Creatinine comes from muscle. \[Click\] Her creatinine of one point one is a normal number for a normal amount of muscle, and she doesn’t have a normal amount of muscle. \[Click\] So the same creatinine probably means a lower true GFR than the fifty-seven the equation returned. Her low muscle mass changes the predictor without changing the truth, and nothing in the equation can tell. \[Click\] Name her model’s predictors, response, and training population. And what do you now say to the attending about the number on the panel? PAUSE: 2 min. Pause here and answer out loud or in writing. \[Click\] Here is one good answer. “The fifty-seven is estimated from her creatinine. With her low muscle mass it probably overstates her kidney function. If the decision depends on it, I would ask for a cystatin C-based eGFR.” Whether it changes the contrast decision depends on how close her true value could be to your department’s threshold. It matters more for a drug dosed by kidney function. The training population was eight thousand two hundred fifty-four adult research participants. Nothing in the equation knows how much muscle she has. Notice also that nobody told her a model had a part in her care. Whether someone should have is a question for the patient communication session. LIVE: Ask for answers in the chat, sent on your count. Cystatin C is the answer you’re looking for, with the caveat that cystatin C has limits of its own and illness can affect it. A measured clearance only if the decision really depends on it.

Every session ends on the same question. What does this change about how you’ll practice? \[Click\] Write your answer down before you close this deck. It is not optional, and one word isn’t an answer. PAUSE: 2 min. Pause here and write it. Then one more. The next time someone shows you a model’s accuracy number, what will you ask in the next two minutes? For example: what was it tested on, and by whom? Some of you’ll be in exactly that spot. A company shows up at lunch with a model that predicts which of your clinic patients will end up in the emergency department, and asks, don’t you want to catch those patients? There are a lot of questions to ask before you say yes, and I hope today’s slides help with some of them. \[Click\] Thursday: we start with the chatbot, the same loop with words. Then we open ChatGPT Edu together and use it to set up your own learning goals and journal.

To sum up. AI contains machine learning, which contains deep learning, which contains generative AI. The eGFR is a model learned from data, with predictors, a response, and a population it was fit to. Fitting chose its numbers to make the error as small as possible. Every learned model has those three parts, and you can ask for them. Supervised learning has the answer in the training data; unsupervised doesn’t. Classification gives a category; regression gives a number. A model that’s too flexible fits the noise. That’s overfitting: a simpler model would have done better on new patients. A neural network is the same loop with far more dials, so it needs more data, and a test set matters even more. A score on the training data describes the fit. It is not evidence. Ask for the test set, and who ran it. Thursday, the same loop with words. The fifty-seven on the panel came from somewhere. So does everything else. LIVE: Skip in a live session.

Try: overfitting you can see **[TensorFlow Playground](https://playground.tensorflow.org)**, 10 minutes, no account. Open [step 1](https://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral®Dataset=reg-plane&learningRate=0.03®ularizationRate=0&noise=50&networkShape=2&seed=0.12345&showTestData=true&discretize=false&percTrainData=20&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false) (too simple), [step 2](https://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral®Dataset=reg-plane&learningRate=0.03®ularizationRate=0&noise=50&networkShape=6,6&seed=0.12345&showTestData=true&discretize=false&percTrainData=20&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification&initZero=false&hideText=false) (in between) and [step 3](https://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral®Dataset=reg-plane&learningRate=0.03®ularizationRate=0&noise=50&networkShape=8,8,8,8,8,8&seed=0.12345&showTestData=true&discretize=false&percTrainData=20&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification&initZero=false&hideText=false) (too flexible). At about 500 epochs, compare **training loss** with **test loss**.

Read: when the patients change **[*The clinician and dataset shift in AI*](https://doi.org/10.1056/NEJMc2104626)**, *NEJM*, 2021. Why models fail on new kinds of patients.

Abràmoff, M. D., Lavin, P. T., Birch, M., Shah, N., & Folk, J. C. (2018). Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. *NPJ Digital Medicine*, *1*, 39.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., et al. (2020). Language models are few-shot learners. *Advances in Neural Information Processing Systems 33 (NeurIPS 2020)*.

Delgado, C., Baweja, M., Crews, D. C., Eneanya, N. D., Gadegbeku, C. A., Inker, L. A., Mendu, M. L., Miller, W. G., Moxey-Mims, M. M., Roberts, G. V., St Peter, W. L., Warfield, C., & Powe, N. R. (2021). A Unifying Approach for GFR Estimation: Recommendations of the NKF-ASN Task Force on Reassessing the Inclusion of Race in Diagnosing Kidney Disease. *American Journal of Kidney Diseases : The Official Journal of the National Kidney Foundation*, *79*(2), 268–288.e1.

Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., Ginneken, B. van, Karssemeijer, N., Litjens, G., Laak, J. A. W. M. van der, Hermsen, M. and, Manson, Q. F., Balkenhol, M., Geessink, O., Stathonikos, N., Dijk, M. C. van, Bult, P., Beca, F., Beck, A. H., Wang, D., Khosla, A., Gargeya, R., … Venâncio, R. (2017). Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. *JAMA*, *318*(22), 2199–2210.

Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. *Nature*, *542*(7639), 115–118.

Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021). The Clinician and Dataset Shift in Artificial Intelligence. *The New England Journal of Medicine*, *385*(3), 283–286.

Halabi, S. S., Prevedello, L. M., Kalpathy-Cramer, J., Mamonov, A. B., Bilbily, A., Cicero, M., Pan, I., Pereira, L. A., Sousa, R. T., Abdala, N., Kitamura, F. C., Thodberg, H. H., Chen, L., Shih, G., Andriole, K., Kohli, M. D., Erickson, B. J., & Flanders, A. E. (2018). The RSNA Pediatric Bone Age Machine Learning Challenge. *Radiology*, *290*(2), 498–503.

Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., & Ng, A. Y. (2019). Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. *Nature Medicine*, *25*(1), 65–69.

Inker, L. A., Eneanya, N. D., Coresh, J., Tighiouart, H., Wang, D., Sang, Y., Crews, D. C., Doria, A., Estrella, M. M., Froissart, M., Grams, M. E., Greene, T., Grubb, A., Gudnason, V., Gutiérrez, O. M., Kalil, R., Karger, A. B., Mauer, M., Navis, G., … Levey, A. S. and. (2021). New Creatinine- and Cystatin C-Based Equations to Estimate GFR without Race. *The New England Journal of Medicine*, *385*(19), 1737–1749.

Levey, A. S., Bosch, J. P., Lewis, J. B., Greene, T., Rogers, N., & Roth, D. (1999). A more accurate method to estimate glomerular filtration rate from serum creatinine: A new prediction equation. Modification of Diet in Renal Disease Study Group. *Annals of Internal Medicine*, *130*(6), 461–470.

Levey, A. S., Stevens, L. A., Schmid, C. H., Zhang, Y. L., Castro, A. F., Feldman, H. I., Kusek, J. W., Eggers, P., Van Lente, F., Greene, T., & Coresh, J. and. (2009). A new equation to estimate glomerular filtration rate. *Annals of Internal Medicine*, *150*(9), 604–612.

Mitchell, T. M. (1997). *Machine learning*. McGraw-Hill.

Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, C. P., Heidenreich, P. A., Harrington, R. A., Liang, D. H., Ashley, E. A., & Zou, J. Y. (2020). Video-based AI for beat-to-beat assessment of cardiac function. *Nature*, *580*(7802), 252–256.

Perou, C. M., Sørlie, T., Eisen, M. B., Rijn, M. van de, Jeffrey, S. S., Rees, C. A., Pollack, J. R., Ross, D. T., Johnsen, H., Akslen, L. A., Fluge, O., Pergamenschikov, A., Williams, C., Zhu, S. X., Lønning, P. E., Børresen-Dale, A. L., Brown, P. O., & Botstein, D. (2000). Molecular portraits of human breast tumours. *Nature*, *406*(6797), 747–752.

Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., Ghous, M., & Singh, K. (2021). External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. *JAMA Internal Medicine*, *181*(8), 1065–1070.

Sodium	139		136–145
Potassium	4.1		3.5–5.1
Creatinine	1.10		0.50–1.10
eGFR (CKD-EPI 2021)	57	L	\>60