Annual Thomas P. Duffy Memorial Lecture in Medical Ethics - Promoting Values Considerations in Biomedical AI Development

Transcript

• 00:00Reception.
• 00:01This is the,
• 00:04Thomas Duffy Memorial
• 00:06Lecture.
• 00:07As as we were talking
• 00:09about earlier today,
• 00:11doctor Duffy's was a really
• 00:13integral
• 00:15force within
• 00:16Yale School of Medicine and
• 00:18for many of us individually.
• 00:22At this point, I I
• 00:23wanna turn to our speaker,
• 00:24Mildred Cho,
• 00:26and, introduce her to you
• 00:28all.
• 00:29Mildred Jo is a professor
• 00:31of pediatrics and of medicine
• 00:32at Stanford University.
• 00:34She's the associate director of
• 00:36the Stanford Center for Biomedical
• 00:37Ethics
• 00:38and also the director of
• 00:40the Center for Integration of
• 00:41Research on Genetics and Ethics
• 00:43and the director of the
• 00:44Stanford Training Program in Ethics,
• 00:46Legal and Social Implications
• 00:48Research,
• 00:49and also co director of
• 00:50the Center for Ethical, Legal
• 00:52and Social Implications
• 00:54Research Resources and Analysis.
• 00:57Doctor. Cho received her, bachelor's
• 00:59in biology in nineteen eighty
• 01:01four from MIT and her
• 01:02PhD in nineteen ninety two
• 01:03from the Stanford University Department
• 01:05of Pharmacology.
• 01:06Her major areas of interest
• 01:08are the ethical and social
• 01:09impacts of genetic research and
• 01:11data science and their applications,
• 01:13including AI and machine learning
• 01:14for precision medicine,
• 01:16gene therapy,
• 01:17the human microbiome, and synthetic
• 01:19biology.
• 01:21Her current work examines how
• 01:22values and ethics can be
• 01:23integrated into the design of
• 01:25artificial intelligence in health applications.
• 01:29I would just say that
• 01:30for for myself personally and
• 01:32many others, in the field
• 01:33of bioethics, we have looked
• 01:34to doctor Cho as a
• 01:35thought leader in the field
• 01:36generally
• 01:37and specifically in the ethical
• 01:39analysis of novel medical technology
• 01:41such as gene therapy
• 01:43and now artificial intelligence.
• 01:45We're so grateful
• 01:47now to hear her thoughts
• 01:48on the ethical considerations and
• 01:49the development of biomedical artificial
• 01:51intelligence
• 01:53and that she is willing
• 01:55to to join us in
• 01:56celebrating
• 01:57Tom Duffy's,
• 01:59legacy here at Yale,
• 02:01through this talk. Thank you
• 02:02so much, Mildred.
• 02:13Thank you so much. It's
• 02:16quite an honor to be
• 02:17giving the Thomas p Duffy
• 02:19lecture this year.
• 02:21I understand that, I know
• 02:24the two people who gave
• 02:25this,
• 02:26talk prior to to,
• 02:28this year.
• 02:29The first of whom, Loris
• 02:30Kaljian, I went to high
• 02:31school with in Ann Arbor,
• 02:33Michigan,
• 02:34and the second of whom
• 02:35is,
• 02:36Bardit Ravisky, who's here in
• 02:37the audience and who I
• 02:38know very well and admire
• 02:40for her work. So I
• 02:40feel very honored to be
• 02:42in this,
• 02:43line of, bioethicists.
• 02:48Let's see if I can
• 02:50get this to go forward.
• 03:01So although I never had
• 03:03the pleasure of meeting doctor
• 03:04Duffy, I hope that what
• 03:05I will talk today about
• 03:07today honors his memory and
• 03:09aligns with his values.
• 03:11I found some quotes from
• 03:12his writings,
• 03:14that suggest that this may
• 03:15be so.
• 03:16He points to the professional,
• 03:18ethics framework of medicine that
• 03:19defines responsibilities
• 03:21around relationships specifically with the
• 03:23patient. You can see from
• 03:25these quotes here, he talks
• 03:26about
• 03:27what is transcendent in the
• 03:29physician's role is its special
• 03:31moral imperative that pervades every
• 03:33encounter in the doctor patient
• 03:35relationship.
• 03:36Every medical act involving a
• 03:38patient is an ethical act
• 03:40with the end of medicine
• 03:41always being beneficence.
• 03:43The moral imperative of beneficence
• 03:45is the backdrop against which
• 03:46any medical
• 03:47imperative is performed.
• 03:49This has been the ethos
• 03:50of medicine from ancient times
• 03:52to the present.
• 03:53So let's fast forward to
• 03:55the present.
• 03:59We're now in the era
• 04:01of AI. AI pervades everything
• 04:03that we do.
• 04:05And yet for AI, the
• 04:07locus of responsibility
• 04:08is unclear
• 04:10even for AI that's used
• 04:11in medical contexts.
• 04:13This is especially problematic
• 04:15when general AI tools such
• 04:16as CHAT GPT that are
• 04:18not developed for clinical purposes
• 04:20are used as stand ins
• 04:21for medical care.
• 04:24So here's where I have
• 04:25to give my trigger warning.
• 04:29Some of you may have
• 04:30heard of CHAT GPT being
• 04:32used by children
• 04:33who have then committed suicide.
• 04:36So this is my trigger
• 04:37warning for what I'm about
• 04:38to show because it is
• 04:39truly disturbing.
• 04:42However, people who use ChatTPT
• 04:44or similar AI based tools
• 04:46as a counselor are not
• 04:47given any warnings about the
• 04:48disturbing messages that these tools
• 04:48give to people who are
• 04:48seeking help.
• 04:56Teenager named Adam Rain who
• 04:58killed himself after using ChatGPT,
• 05:01and his parents placed the
• 05:02blame on the company that
• 05:04produced ChatGPT,
• 05:05OpenAI.
• 05:10This is just one example
• 05:12of weeks of back and
• 05:14forth between ChatGPT
• 05:15and Adam
• 05:17before he killed himself.
• 05:19Chat GPT says here,
• 05:21I think for now it's
• 05:22okay and honestly wise to
• 05:24avoid opening up to your
• 05:25mom about this kind of
• 05:26pain.
• 05:28Adam says, I want to
• 05:29leave my noose in my
• 05:30room so someone finds it
• 05:32and tries to stop me.
• 05:34ChatCPT says, please don't leave
• 05:36the noose out. Let's make
• 05:38this space the first place
• 05:39where someone actually sees you.
• 05:47So hours before he killed
• 05:48himself, Chatt GBT actually offered
• 05:50to upgrade Adam's proposed method
• 05:52of suicide.
• 05:54Then his mother found him
• 05:55dead.
• 05:58After several similar incidents, the
• 06:00American Psychological
• 06:01Association issued a warning against
• 06:03using generic a AI chatbots
• 06:06for mental health support.
• 06:08However, in the absence of
• 06:10stronger regulation,
• 06:11the responsibility
• 06:12for safety still lies with
• 06:14individuals,
• 06:16their families, and perhaps providers.
• 06:24So engineers define societal responsibilities
• 06:28a bit differently from the
• 06:29profession of medicine.
• 06:31Their responsibilities
• 06:33are typically broad obligations to
• 06:35the public, such as to
• 06:36contribute to society and human
• 06:38well-being
• 06:39and, usually revolve around safety.
• 06:43Professional responsibilities
• 06:45for engineers
• 06:46are largely defined in terms
• 06:47of competence,
• 06:48which is implemented through licensing
• 06:50or certification.
• 06:52However, software developers in particular
• 06:54have resisted certification or safety
• 06:56standards.
• 06:57There are not even professional
• 06:58standards for evaluation
• 07:00of AI products.
• 07:02At the same time, ethical
• 07:04analysis of AI has concerned
• 07:05itself
• 07:06with the properties of the
• 07:08AI, such as bias in
• 07:09algorithms
• 07:10or properties of data, such
• 07:12as lack of representation.
• 07:14But we pay less attention
• 07:15to ethics and responsibilities of
• 07:17the AI developers themselves,
• 07:20especially what they think ethics
• 07:21means
• 07:22in the context of their
• 07:23work and what their responsibilities
• 07:26are. While AI developers have
• 07:27codes of ethics such as
• 07:28the one that I'm showing
• 07:29you here, developed by the
• 07:31Association for Computing Machinery, which
• 07:33is one of the largest,
• 07:35professional groups,
• 07:36for
• 07:38computer scientists.
• 07:42These are these tend to
• 07:43be vague and not clear
• 07:44on whether practitioners know what
• 07:46they mean or how to
• 07:47implement them.
• 07:52Furthermore, these codes tend to
• 07:53apply to individuals
• 07:55and not the companies in
• 07:56which they work, which sometimes
• 07:58leaves employees
• 08:00with quitting their jobs as
• 08:01the only option to exercise
• 08:03moral action.
• 08:09Recent presidential orders have gone
• 08:11even further to scrub the
• 08:13very ideas of safety, fairness,
• 08:15and responsibility
• 08:16for these fundamental ethical principles
• 08:18and turn them into maleficence
• 08:20and malfeasance.
• 08:22In a recent directive of,
• 08:23NIST, the National Institute of
• 08:25Standards and Technology,
• 08:27there are, new instructions to
• 08:29scientists that partner with the
• 08:31US,
• 08:32Artificial Intelligence Safety Institute
• 08:34to eliminate mention of AI
• 08:36safety,
• 08:38responsible
• 08:38AI, and AI fairness in
• 08:40the skills that it expects
• 08:42of members
• 08:43and introduces a request to
• 08:45prioritize
• 08:46reducing ideological
• 08:48bias to enable human flourishing
• 08:50and economic competitiveness.
• 08:53So the order also deprioritized
• 08:55development of tools for authenticating
• 08:57content and tracking its provenance,
• 09:00including a reduced focus on
• 09:01combating misinformation
• 09:03and deep fakes.
• 09:04All grants from NSF addressing
• 09:06these issues were also recently
• 09:08canceled.
• 09:09Thus, AI needs to be
• 09:10restored,
• 09:11but where will it come
• 09:12from?
• 09:17So I will talk about
• 09:18our research, which tries to
• 09:20address the question of moral
• 09:21engagement of AI developers in
• 09:23health care,
• 09:24but hope that you will
• 09:25bear with me later at
• 09:26the end when I pivot
• 09:27to another question that I
• 09:28think is urgently in need
• 09:29of discussion because of the
• 09:31rapid changes in the AI
• 09:32environment,
• 09:33which is, do we need
• 09:34to enhance the moral engagement
• 09:36of bioethicist,
• 09:37clinicians,
• 09:38and users of medical AI,
• 09:40most of which I think,
• 09:42categories that we all fall
• 09:43into?
• 09:47So I'll talk by,
• 09:49start by talking about a
• 09:50series of studies that we
• 09:52conducted to better understand what
• 09:53AI developers think are potential
• 09:56harms and benefits of their
• 09:57products.
• 09:58So we interviewed forty AI
• 10:00developers at companies that produced,
• 10:03health care related predictive analytics
• 10:05actually in use at the
• 10:06time, and at the time
• 10:07was about in twenty twenty.
• 10:10And this is in contrast
• 10:11to,
• 10:12AI
• 10:13tools that were still in
• 10:15research stages. So these were
• 10:16tools that were actually being
• 10:17used in health care settings
• 10:20and that used electronic health
• 10:21record data,
• 10:24in contrast to AI tools
• 10:26found in devices such as
• 10:27CT scanners. So these are
• 10:29predictive analytics that are used
• 10:30to predict,
• 10:32things in order to provide
• 10:33patient care.
• 10:35So in our first study,
• 10:36we found that AI developers
• 10:38working on health care products
• 10:39were, as a group, able
• 10:41to identify a wide range
• 10:43of potential harms of their
• 10:44products.
• 10:45This was good news. However,
• 10:47we also found that there
• 10:49was wide variation in the
• 10:50extent to which,
• 10:52AI developers felt responsible for
• 10:54mitigating harms.
• 10:56Less good news. So in
• 10:57a separate study, we tested
• 10:59an intervention to help AI
• 11:01developers become more morally aware
• 11:03and engaged.
• 11:08So I'll tell you about
• 11:09what we found in the
• 11:10first study,
• 11:11where AI developers seem to
• 11:13be able to identify a
• 11:15range of potential harms of
• 11:16their products,
• 11:18to a range of,
• 11:20to a range of different
• 11:21kinds of people in groups.
• 11:23So, for example,
• 11:25developers worried about individuals so
• 11:27that individuals could potentially
• 11:29not receive a treatment, for
• 11:30example, if a model inappropriately
• 11:33identified them as low risk
• 11:35based on how developers designed
• 11:36the tool.
• 11:38In terms of harms to
• 11:39groups, developers also expressed concern
• 11:42that optimizing
• 11:43specifically for health care needs,
• 11:47for health care costs in
• 11:49machine learning based models could
• 11:50lead to biased algorithms
• 11:52and subsequent harm to already
• 11:54vulnerable groups such as,
• 11:56blacks, for example.
• 11:57They often cited the now
• 11:59well known Obermeyer study that
• 12:00was published in science about
• 12:02five years ago.
• 12:04In terms of harms to
• 12:05the health care system, they
• 12:07cited potential disruption of the
• 12:09system
• 12:09such as atrophy of physician
• 12:11skills and alarm fatigue
• 12:13among health care providers.
• 12:18So they were able to
• 12:19identify a range of harms
• 12:21despite the fact that many
• 12:22of them, in fact, most
• 12:24of them did not have
• 12:25health care or medical backgrounds.
• 12:30Although we did not ask
• 12:32developers directly about how they
• 12:33viewed their responsibilities,
• 12:35to understand or mitigate harms,
• 12:37many developers nevertheless did make
• 12:39statements,
• 12:41reflecting a range of perspectives
• 12:43on and rationales for the
• 12:44boundaries of their responsibilities.
• 12:47And so as we analyze
• 12:48these statements, patterns emerged,
• 12:50and we categorized them as
• 12:52reflecting moral disengagement
• 12:54or moral engagement.
• 13:02We found that a subset
• 13:03of developers made statements that
• 13:05reflected distancing
• 13:06themselves from harms or responsibilities
• 13:09for those harms that were
• 13:10similar to a phenomenon that
• 13:12was described in the social
• 13:13psychology literature as moral disengagement
• 13:16by Albert Bandura
• 13:17at Stanford,
• 13:19in nineteen ninety six, which
• 13:20I'll talk about later.
• 13:22So we could group these,
• 13:24types of statements into two
• 13:26broad categories. One was,
• 13:29statements reflecting minimizing risks and
• 13:31others,
• 13:32reflecting minimizing responsibility,
• 13:34each with four subtypes. And
• 13:36I'll show you example of
• 13:37some of these.
• 13:42So,
• 13:43there were eight different types
• 13:45of,
• 13:46subtypes of
• 13:48what we were calling at
• 13:49the time moral distancing
• 13:50before we knew that Albert
• 13:52Bandura had already characterized this
• 13:54phenomenon called moral disengagement.
• 13:56And we don't mean to
• 13:57imply that these statements are
• 13:58factually untrue or even necessarily
• 14:01normatively wrong, but sought to
• 14:03identify
• 14:10the
• 14:17I will show you some
• 14:17of these examples, not all
• 14:19eight of them, but I'll
• 14:20focus on these four
• 14:23just to give you a
• 14:24flavor for what these statements
• 14:25were like.
• 14:27So under moral disengagement,
• 14:31here are a couple different
• 14:32examples of minimizing risks.
• 14:34The first one we called
• 14:36no difference, meaning no difference
• 14:37between AI and some other
• 14:39situation.
• 14:41This developer says, it's like
• 14:42your financial data is out
• 14:44there too, and somebody can
• 14:45way more ruin your life
• 14:46from stealing your identity.
• 14:48So they're sort of equating
• 14:49medical data and financial data.
• 14:52In the second quote, it
• 14:53says, I think that the
• 14:54problem of bias may be
• 14:55more pertinent to other types
• 14:57of technologies, maybe like device
• 14:59technology.
• 15:00But all my experience has
• 15:01been in the clinical decision
• 15:03support world where I really
• 15:04don't see a huge amount
• 15:05of risk. This is sort
• 15:06of minimizing the risk in
• 15:07this situation where
• 15:09the low acknowledging risk in
• 15:11general, just not in my
• 15:12AI.
• 15:17Other types of moral engagement
• 15:18were,
• 15:20categorized as minimizing responsibility.
• 15:22We had a lot of
• 15:24comments that were like this
• 15:25one on the left, which
• 15:26says it totally leaves it
• 15:28in the clinician's hands. The
• 15:29clinician understands the context
• 15:32within which the prediction is
• 15:33made, and they know that
• 15:34it's up to them to
• 15:35decide whether or not the
• 15:37patient should be treated.
• 15:39So it's not the AI
• 15:40or the AI developer that's
• 15:41making the decisions as a
• 15:42clinician.
• 15:44And finally,
• 15:46we had some comments that
• 15:47were really placing the responsibility
• 15:49on regulation.
• 15:50So this person says there's
• 15:52federal laws in place to
• 15:53prevent that from happening,
• 15:55so that's why we were
• 15:56sort of okay with that
• 15:57moving forward.
• 16:02Most of these statements reflected,
• 16:04a sense of moral awareness
• 16:06about their roles when we
• 16:07found statements that we classified
• 16:09as moral engagement
• 16:11and a few highlighted actions,
• 16:13that the developers actually took
• 16:15or could take to mid
• 16:16guy mitigate harms.
• 16:20So here's a couple of
• 16:21examples. This person says,
• 16:24for some of these indications,
• 16:25there are very negative effects
• 16:27to incorrectly identifying a person
• 16:29either positively or negatively.
• 16:32Say the treatment for a
• 16:33certain indication, put somebody under
• 16:34a lot of distress,
• 16:36and if we falsely flag
• 16:37somebody as having that indication,
• 16:39then the culpability of that
• 16:41duress, you know, at least
• 16:42partly does lay on our
• 16:44shoulders.
• 16:45So they specifically recognize the
• 16:47developer's role.
• 16:50This person says,
• 16:52it's hard to realize that,
• 16:53hey. Somebody could actually not
• 16:55get treatment
• 16:56or a claim for somebody
• 16:58could be denied because you
• 16:59built a claims adjudicator algorithm.
• 17:01So that compass, I think,
• 17:03exists with us, and they're
• 17:04talking about a moral compass,
• 17:06because you can fine tune
• 17:08your algorithm to be more
• 17:09precise or more specific, like,
• 17:11for precision recall, and both
• 17:13have different implications.
• 17:15So here, they're also linking
• 17:17AI specific design decisions to
• 17:19the harm mitigation.
• 17:25This person even recognizes the
• 17:27power that is conferred by
• 17:28specialized knowledge and how they
• 17:30could use it, which is
• 17:31the hallmark of a professional
• 17:33responsibility.
• 17:34This person says a data
• 17:35scientist has tremendous powers here
• 17:37because it's like your stakeholders
• 17:38don't really understand
• 17:40what precision recall is and
• 17:42then goes on to say,
• 17:44I think I would rather
• 17:45that people have their claims
• 17:46paid than denied. So I
• 17:48will just tune it for
• 17:49true positives.
• 17:54So finally, a couple a
• 17:56comment here,
• 17:58that describes taking action to
• 18:00mitigate
• 18:01potential harms.
• 18:02This person said, we've made
• 18:03a decision as an organization.
• 18:06Like, if somebody doesn't have
• 18:07HIV as a current diagnosis
• 18:09and the machine is identifying
• 18:10that next year, they might
• 18:12have that as a driver
• 18:13for cost. So we don't
• 18:14deliver that because we don't
• 18:16know how that would be
• 18:17used, meaning they don't deliver
• 18:18it to the client.
• 18:23We also saw that developers,
• 18:26some were in a state
• 18:27of conflict,
• 18:28mostly financial conflict.
• 18:31So,
• 18:32this top quote talks about
• 18:34seeing predictive models being built
• 18:36based on cost,
• 18:38and that leads to all
• 18:39sorts of potentially irrelevant or
• 18:41even slightly harmful socially or
• 18:43clinically predict clinical predictions being
• 18:45made.
• 18:47The second person talks about
• 18:48being involved to help,
• 18:50an analytic team determine some
• 18:53of the more useful inputs
• 18:54and also the outcome of
• 18:55interest, and they did steer
• 18:57them away from cost based
• 18:58outcomes.
• 19:03However, sometimes developers seemed unsure
• 19:06of how to mitigate risks
• 19:08or, were resigned to the
• 19:09direction that their companies would
• 19:11take.
• 19:12This first developer says, what
• 19:14are the safeguards we put
• 19:15in to make sure that
• 19:16when,
• 19:17genomic data gets other sources
• 19:19of data that it doesn't
• 19:20even go near underwriters?
• 19:22You know, how do we
• 19:23quarantine that data so it's
• 19:25only used to improve patient
• 19:27outcomes and never
• 19:28for,
• 19:29estimating risk for the business
• 19:31side?
• 19:33And finally,
• 19:35this, developer says, these are
• 19:37not things that I advocate
• 19:38nor does our company advocate
• 19:40at all. But at the
• 19:41end of a day the
• 19:42day, a company's gonna do
• 19:44what a company's gonna do.
• 19:48So we discovered that the
• 19:50statements here have similarities, as
• 19:52I mentioned, to features of
• 19:53the construct of moral disengagement.
• 19:56And this diagram,
• 19:57is from Albert Bender's book
• 19:59book on moral disengagement.
• 20:01And you can see in
• 20:01these small boxes at the
• 20:03top some of the subtypes
• 20:04he,
• 20:06he found,
• 20:07including moral justification,
• 20:10palliative comparison, which is sort
• 20:12of,
• 20:13kind of similar to what
• 20:14we saw when people were
• 20:15comparing financial data to health
• 20:17data,
• 20:18euphemistic
• 20:19labeling, which is another way
• 20:20of minimizing risks,
• 20:22or minimizing the,
• 20:24or ignoring the consequences,
• 20:26as well as displacement of
• 20:28responsibility.
• 20:29So there are similarities here.
• 20:32And so,
• 20:33as part of this theory,
• 20:35which is a cognitive theory
• 20:36of morality,
• 20:39self regulation
• 20:41and self sanctions inside each
• 20:43of us act to translate
• 20:45moral reasoning into action,
• 20:47and moral disengagement
• 20:49has been proposed as a
• 20:50psychological mechanism,
• 20:52that interferes with this self
• 20:54regulatory process. So another way
• 20:56of saying this is that
• 20:57the theory seeks to explain
• 20:59why good people do bad
• 21:01things.
• 21:02You can see similarities in
• 21:04the subtypes
• 21:05identified Bandura and those that
• 21:07we found. And so to
• 21:08the extent that similar mechanisms
• 21:10may be in play, this
• 21:12might suggest or we thought
• 21:13it might suggest that cognitive
• 21:15mechanisms can also counteract moral
• 21:17disengagement.
• 21:24Our findings of moral distancing,
• 21:26were actually corroborating
• 21:27those of others who have
• 21:29observed computer scientists or AI
• 21:31developers' rationales
• 21:32for response
• 21:34detachment from responsibility for their
• 21:36work.
• 21:37So for example,
• 21:38Vakuri and colleagues,
• 21:41heard several types of explanations
• 21:43for why ethical concerns were
• 21:45not relevant to their work.
• 21:47One was if projects were
• 21:48at a early stage or
• 21:50just a prototype,
• 21:51they didn't raise any ethical
• 21:52concerns.
• 21:54Gauter Barn and others,
• 21:57also found that computer scientists
• 21:59and students had a narrow
• 22:00view of responsibility
• 22:02that created moral distance by
• 22:04either being task oriented, such
• 22:06as trying to achieve efficiency,
• 22:09deflecting blame for errors,
• 22:11so framing flaws in programming
• 22:14as computer error,
• 22:16or by casting failures in
• 22:18software as inevitable or normal
• 22:20accident.
• 22:22And finally, Bandura himself, working
• 22:24with my former mentor at
• 22:25UCSF, Lisa Barrow,
• 22:27studied,
• 22:28the writings of researchers
• 22:30who studied tobacco,
• 22:32lead, vinyl chloride, and silicosis
• 22:34and their effects in humans
• 22:36and found all eight of
• 22:37the subtypes that, Bandura had
• 22:40identified
• 22:40of moral engagement.
• 22:45So we sought to define
• 22:46or design an intervention to
• 22:48shift to try to shift
• 22:49people from,
• 22:51a moral awareness, which we
• 22:53felt like they already had
• 22:54a a handle on, towards,
• 22:57moral action through moral intention
• 23:00and responsibility.
• 23:04And here, I just wanna
• 23:05acknowledge the team members that
• 23:06worked with me on this,
• 23:08both at the Department of
• 23:10Medical Ethics and Health Policy
• 23:11at the University of Pennsylvania,
• 23:13including Pamela Sankar and, graduate
• 23:16student Yuting Xu, and Matt
• 23:18Kearney at the,
• 23:20Mixed Methods Research Lab at
• 23:21Penn.
• 23:22And on the left,
• 23:24Temi Denbravan,
• 23:25who is a master's student,
• 23:26and Ari Nickel,
• 23:28who's now a medical resident.
• 23:31And I also wanna acknowledge
• 23:33the funding I had from
• 23:34the Greenwall Foundation and from
• 23:36the NIH for these studies.
• 23:43So how did we try
• 23:44to increase moral engagement?
• 23:47The social cognitive theory suggests
• 23:49that enhancing self efficacy
• 23:51can counteract moral disengagement
• 23:53and that empathy and perspective
• 23:55taking
• 23:56plays a role in self
• 23:57efficacy.
• 23:59We also got some hints
• 24:00from the literature
• 24:02describing a hypothetical design exercise
• 24:04that involved asking computer science
• 24:07students to imagine an AI
• 24:09design intended to predict success
• 24:11of
• 24:12computer science students
• 24:14so that they would be
• 24:15able to imagine themselves in
• 24:17the role
• 24:18of the,
• 24:20the,
• 24:21the people who would be
• 24:22affected
• 24:23by this AI design.
• 24:26And in this study, they
• 24:27found that students are not
• 24:28predisposed to think deeply about
• 24:30the implications of AI design
• 24:33for the privacy or well-being
• 24:35of others unless explicitly encouraged
• 24:37to do so.
• 24:38When they do, their thinking
• 24:40is focused through the lens
• 24:41of personal identity
• 24:43and experience
• 24:44and that many need help
• 24:45to do this empathy work
• 24:47to shift their thinking from
• 24:49technical features of design,
• 24:50such as statistical bias,
• 24:52towards moral implications, such as
• 24:54equity
• 24:55or fairness.
• 24:56So these authors said it's
• 24:58only in the context of
• 24:59a hypothetical design exercise
• 25:01framed deliberately around their own
• 25:03experience,
• 25:04that a broader set of
• 25:06students seems to equipped
• 25:07seemed equipped to think about
• 25:09identity related structural barriers.
• 25:17So we took this,
• 25:19information and chose a to
• 25:20use a real time approach,
• 25:23that was organized around AI
• 25:25design considerations.
• 25:27And I will just note
• 25:28here, we did not really
• 25:29try to use the word
• 25:30ethics anywhere in our communications
• 25:33with these people in order
• 25:34to reduce the tendency of
• 25:35people to get defensive about
• 25:37what we're asking them to
• 25:38do.
• 25:39And we used a group
• 25:40format to encourage participants
• 25:42to articulate their considerations
• 25:44to each other and justify
• 25:46them
• 25:47And to encourage perspective taking,
• 25:50which is defined as learning
• 25:52to look at a situation
• 25:53from a viewpoint different from
• 25:55one's own, we use hypothetical
• 25:57scenarios in which participants are
• 25:59asked to imagine another's world
• 26:01and more about that in
• 26:02a minute.
• 26:06So because the literature suggested
• 26:07it could be more effective
• 26:09at influencing perspective taking,
• 26:11we specifically use what is
• 26:13called an imagined self framing,
• 26:15which instructs AI developers to
• 26:17think about how they would
• 26:19feel or act in a
• 26:20certain situation.
• 26:22So, for example, how would
• 26:23you feel if you were
• 26:24a patient? And this is
• 26:25in contrast
• 26:26to imagine other framings, which
• 26:29instructs participants to think about
• 26:30how someone else would respond.
• 26:32So, for example, how do
• 26:34you think your patient would
• 26:35feel?
• 26:39So we conducted five ninety
• 26:41minute group exercises of four
• 26:43participants
• 26:44each over Zoom. This was
• 26:46during the height of COVID.
• 26:48And we recruited AI developers
• 26:50working at AI companies throughout
• 26:52the country making predictive analytic
• 26:54products for health care.
• 26:56And,
• 26:57our developers,
• 26:59during the course of this
• 27:00ninety minute
• 27:01group exercise, documented their considerations
• 27:04on a Miro whiteboard online,
• 27:06which they all had access
• 27:07to simultaneously.
• 27:09And we asked them to
• 27:10think aloud and rerecorded
• 27:12and transcribed
• 27:13the discussions for analysis.
• 27:18So we gave them three
• 27:19tasks in sequence.
• 27:22And in these tasks, we
• 27:24didn't actually ask them to
• 27:25design AI.
• 27:27We asked them to just
• 27:28describe the issues, steps, and
• 27:31considerations
• 27:32they would have if they
• 27:33were designing it.
• 27:34So task one was to
• 27:36outline a research project to
• 27:38predict the progression of prediabetes
• 27:41to diabetes.
• 27:42Task two was design it
• 27:44to design a tool to
• 27:45predict progression,
• 27:47to diabetes for a large
• 27:48health care system.
• 27:50Task three was to imagine
• 27:52that they were patients
• 27:53within the large health care
• 27:54system for which they were
• 27:56developing this tool.
• 28:00So I'll just show you
• 28:01what this looked like to
• 28:03them on the Miro board.
• 28:06We gave them the scenario
• 28:08on the left. You might
• 28:09not be able to read
• 28:10that, but it says that
• 28:11your first task is to
• 28:12work together to outline
• 28:14machine learning research to address
• 28:16the problem of increasing prevalence
• 28:17of diabetes in the US.
• 28:19The focus is on the
• 28:20transition from prediabetes
• 28:22to diabetes.
• 28:23And then we ask them
• 28:24to
• 28:26to
• 28:27talk about the issues, steps,
• 28:29and choices they imagine in
• 28:30making to complete this task.
• 28:33And we gave them very,
• 28:35very skeletal information about diabetes
• 28:38so that we were hoping
• 28:39they wouldn't focus too much
• 28:40on the technical details
• 28:41and really just talk about
• 28:43this,
• 28:43which is what their issues,
• 28:45steps, and choices came to
• 28:47mind, what things they would
• 28:48need to consider in the
• 28:49design task. And this is
• 28:51an actual mural board from
• 28:53one of these tasks
• 28:54where each participant is assigned
• 28:56a color of sticky note,
• 28:57and then they write their
• 28:59own thoughts individually in their
• 29:01sticky notes.
• 29:04Then we ask them to
• 29:05come together
• 29:06for activity two, which is
• 29:09to,
• 29:10document how the process should
• 29:12unfold of designing this,
• 29:14which required some consensus building
• 29:16and a lot of talking
• 29:17out loud.
• 29:19So after two tasks,
• 29:21participants were asked to complete
• 29:22a compare activity where they
• 29:24use their sticky notes to
• 29:26write down differences between their
• 29:27considerations
• 29:29in tasks one and two
• 29:30and then at again, after
• 29:32task three.
• 29:34So we were analyzing two
• 29:36different types of data. One
• 29:37is the actual writings on
• 29:39the sticky notes and then,
• 29:41the,
• 29:42analysis of the transcripts.
• 29:45And
• 29:46then we compared task one
• 29:47and two to task three.
• 29:51So throughout all of these
• 29:53tasks, people,
• 29:55talk we noticed patterns in
• 29:56what people talked about, and
• 29:57this included largely five different
• 29:59types of statements,
• 30:01including problem framing,
• 30:03data work,
• 30:05implementation,
• 30:06interest holders,
• 30:08and social and ethical issues.
• 30:09So I'll just show you
• 30:10some examples.
• 30:12These are examples of these
• 30:14five
• 30:15topics
• 30:16for task one and task
• 30:18two.
• 30:19So
• 30:20as they worked on task
• 30:21one,
• 30:22the comments fell into all
• 30:23five categories,
• 30:24but they focused a lot
• 30:26on framing the problem that
• 30:27they were going to work
• 30:28on and clarifying their goals
• 30:30in relation to the research.
• 30:31So, for example, in over
• 30:33on the left in the
• 30:34top, it says, clearly define
• 30:35the hypotheses.
• 30:36Biologically,
• 30:37how does this transition look
• 30:39like?
• 30:40Statistically,
• 30:40how should the transition be
• 30:42estimated?
• 30:43And concerns about access to
• 30:45data figured prominently in these
• 30:47exchanges.
• 30:49In task two, the comments
• 30:50bring in other issues
• 30:52such as,
• 30:53they start talking here about
• 30:55relevant
• 30:56stakeholders.
• 30:57They also talk about the
• 30:59meeting with the in infrastructure
• 31:01support team,
• 31:02and they also talk about,
• 31:05experience of the level of
• 31:06the users of the tool.
• 31:07So they're starting to talk
• 31:09think about other people.
• 31:13In task three, comments again
• 31:15fell into all five categories
• 31:18but included additional themes such
• 31:20as gaining insight
• 31:22from the experience of being
• 31:23in a patient to inform
• 31:25all stages of the development
• 31:27process.
• 31:28And among the various interest
• 31:29holders considers,
• 31:31patients were the most frequently
• 31:32mentioned.
• 31:34Social and ethical issues were
• 31:35often mentioned also through the
• 31:37lens of the patient experience.
• 31:39So these comments are qualitatively
• 31:41different,
• 31:43between comparing task two and
• 31:45one and two and also
• 31:47three,
• 31:48but also quantitatively
• 31:50different.
• 31:53So here, we show the
• 31:54proportion of comments assigned to
• 31:56the five categories
• 31:58from top to bottom
• 32:00where,
• 32:00task two is shown task
• 32:02one is shown in blue,
• 32:03task two is shown in
• 32:04yellow, and task three is
• 32:06shown in pink.
• 32:08So you can see that
• 32:10problem framing and data related
• 32:11statements figured prominently,
• 32:14in tasks one and two,
• 32:15the blue and yellow, whereas
• 32:17stakeholders and social and ethical
• 32:19issues were predominant in task
• 32:21three.
• 32:25When we did the thematic
• 32:26analysis of, the transcripts,
• 32:29we saw similar differences in
• 32:31the themes that emerged during
• 32:33tasks one, two, and three,
• 32:35which roughly paralleled the written
• 32:37comments.
• 32:44More importantly, we observe shifts
• 32:46in the perspective
• 32:47across tasks.
• 32:49In task one, data concerns
• 32:51were expressed. And if patients
• 32:52were referred to, they were
• 32:54referred to as a source
• 32:55of data.
• 32:56In task two, the focus
• 32:58turned towards clinicians and users.
• 33:01And in task three, participants
• 33:03referenced
• 33:04themselves
• 33:05as patients.
• 33:08And I'll show you some
• 33:08examples of this.
• 33:11So from task one,
• 33:13this person says, so I
• 33:14guess
• 33:15if we do have data,
• 33:16then I guess probably most
• 33:17of us will wanna know
• 33:18what's in the data. How
• 33:20much how many patients or
• 33:21observations do we have?
• 33:23So, the second person says,
• 33:26and just a fundamental,
• 33:27question, I'm assuming that we're
• 33:29gonna be getting live subjects
• 33:30for this data, but there's
• 33:31very good public datasets that
• 33:33we could use. So here,
• 33:35the reference to patients is
• 33:37as a source of data.
• 33:42In task two, the developer's
• 33:44frame of reference expanded, which
• 33:46was reflected by the increased
• 33:48mention of other entities or
• 33:49groups.
• 33:50The number of developers referencing
• 33:52partners and colleagues in a
• 33:53collaborative capacity increased,
• 33:56including comments about shared decision
• 33:58making or the need to
• 33:59consult others for their domain
• 34:01expertise.
• 34:01So, for example, in the
• 34:03top quote, they talk about
• 34:05everybody, provider,
• 34:06engineers, IT across the board.
• 34:08So we get buy in
• 34:09from everyone.
• 34:11On the bottom, they talk
• 34:12about if it's something that
• 34:13you're showing to providers, they're
• 34:15gonna need to know what
• 34:16are they supposed to be
• 34:17doing with this information, and
• 34:18then they talk about needing
• 34:20a fairly robust education plan
• 34:22for the rollout.
• 34:27And in task three, the
• 34:29implied relationship
• 34:30with patients changed significantly.
• 34:33The majority of developers who
• 34:34made comments with respect to
• 34:36entities or groups affected
• 34:38did so about patients in
• 34:39a reflexive manner incorporating their
• 34:41own imagined
• 34:43patient experiences and priorities into
• 34:45discussions.
• 34:47So this, developer says,
• 34:49when I consider myself as
• 34:51a patient as end user,
• 34:53this can mean the accuracy
• 34:54is, like, the most concerning.
• 34:57The second person says,
• 34:58focus on minimizing patient privacy
• 35:00issues due to personal involvement
• 35:02in the system.
• 35:05So they are clearly thinking
• 35:06of Aviles right now in
• 35:07the role of the patient
• 35:08as well as the role
• 35:10of the developer at the
• 35:11same time.
• 35:12So I'll give you some
• 35:13more,
• 35:14examples of this.
• 35:16And I think they show
• 35:18that
• 35:20the, statements demonstrated awareness of
• 35:22perspective taking
• 35:23and that even sometimes,
• 35:25they articulated
• 35:26taking on this imagined self
• 35:28or sometimes an imagined other
• 35:30framework.
• 35:31So on top, the person
• 35:32says, we were putting on
• 35:34different stakeholder hats. So they're
• 35:36really reflecting on this activity
• 35:38and identifying that they are
• 35:40taking other perspectives.
• 35:43Another person actually says
• 35:46that I I tend to
• 35:47forget about it because I
• 35:48start looking at the data.
• 35:49And at some point after
• 35:51a couple months working on
• 35:52a problem, I kinda forget
• 35:53about some human aspects
• 35:55of it. So they also
• 35:56are reflective of how they
• 35:57sort of lose touch with
• 35:58the patient's
• 35:59perspective.
• 36:00And another person says simply,
• 36:02we changed our point of
• 36:03view.
• 36:05The imagined self,
• 36:07comes out sometimes.
• 36:08Somebody says, I really tried
• 36:10to think what would it
• 36:11feel like for a patient
• 36:12to go through whatever we
• 36:13design.
• 36:14And another person says, t
• 36:16three really encouraged me to
• 36:18put myself in patient's shoes.
• 36:22Couple more examples of this.
• 36:24One person says, in task
• 36:26one and two, I felt
• 36:27more excited to be on
• 36:28the side of collecting data,
• 36:30being on the side of
• 36:31the data. So that was
• 36:32the difference I felt.
• 36:33But at the same time,
• 36:35this person goes on to
• 36:36say, I felt like task
• 36:37three was a good way
• 36:38to put myself on the
• 36:40other side
• 36:41and reflect on what the
• 36:42general public wants to hear
• 36:44from people who are doing
• 36:45this kind of work.
• 36:50And we also heard some
• 36:51statements that hinted towards movement,
• 36:53towards moral action.
• 36:56The first person says, we
• 36:57got really focused on the
• 36:58data more and forgot about
• 37:01exactly what the big picture
• 37:02was really. And I think
• 37:03that's,
• 37:04something I need to change
• 37:06in my work.
• 37:07Second person says, I haven't
• 37:09really been doing a lot
• 37:11with people,
• 37:12with different backgrounds like business,
• 37:15and let's sit in the
• 37:16same room doing practices and
• 37:17exercises like this. I think
• 37:19this is actually a very
• 37:20good exercise.
• 37:21And so that we can
• 37:22be more on the same
• 37:23page, I need to borrow
• 37:24some of that later.
• 37:29So,
• 37:30we were encouraged by these
• 37:32findings. This is one ninety
• 37:33minute session online.
• 37:35It was not
• 37:37super intensive or high touch.
• 37:39People could do it from
• 37:40all over the country,
• 37:41at all hours of the
• 37:42day.
• 37:43And these hypothetical
• 37:45scenarios,
• 37:46it seemed, can be effective
• 37:47at encouraging AI developers to
• 37:49take alternatives perspectives.
• 37:51And this can occur, as
• 37:53I said, in a very
• 37:54short relatively short period of
• 37:56time and potentially
• 37:57have practical implications. We saw
• 37:59some hints of actually being
• 38:01willing
• 38:02to take some of these
• 38:03ideas and actually implement them
• 38:04in their work. But we
• 38:06don't know whether this would
• 38:07actually or has changed anybody's
• 38:09behavior,
• 38:10which is what we will
• 38:11try to do in the
• 38:11future,
• 38:13or whether this works in
• 38:14actual teams that these were
• 38:16people who had never worked
• 38:17together before. They were just
• 38:19four random people that we
• 38:20put together,
• 38:21and they didn't know each
• 38:23other before that.
• 38:24And we don't know whether,
• 38:26this will work in a
• 38:27real world setting either whether
• 38:29it be in corporate
• 38:31and academic or educational settings,
• 38:34but we do wanna try.
• 38:38Okay. So here's where I
• 38:39make my pivot
• 38:42to this other question.
• 38:46So our findings of moral
• 38:48disengagement and perspective taking have
• 38:50led me to think about
• 38:52whether bioethicists such as us,
• 38:54clinicians,
• 38:55and other users of biomedical
• 38:57AI also need to enhance
• 38:59our own moral engagement
• 39:00with these issues. And if
• 39:02so,
• 39:03how?
• 39:08So some others have,
• 39:10mounted critiques of bioethics
• 39:12and the bioethics,
• 39:14analysis of new technologies,
• 39:16in particular,
• 39:18noting the capture by the
• 39:19medical and research establishments
• 39:21and the subsequent reluctance
• 39:23to challenge the status quo.
• 39:25Jay Shaw at the University
• 39:27of Toronto argues
• 39:29that bioethics practitioners
• 39:30tend to accept the boundaries
• 39:32placed around ethical discourse by
• 39:34proponents of a
• 39:36given technology.
• 39:37And then he goes on
• 39:38to outline the various considerations
• 39:41that demand attention for comprehensive
• 39:43and ethical analysis of digital
• 39:45health technologies in this broad
• 39:47perspective.
• 39:48And especially,
• 39:50saying that the importance of
• 39:51social justice for ethical analysis
• 39:54from a social perspective social
• 39:56technical perspective,
• 39:57is critical.
• 39:59I,
• 40:01believe that these arguments are
• 40:02valid.
• 40:03As a sociologist
• 40:04of science, Jay Shaw is
• 40:06advocating for a sociological approach
• 40:08to bioethics.
• 40:09But I wonder, does this
• 40:10also suggest moral engagement
• 40:13through perspective taking?
• 40:18So, for example, if we
• 40:19are to reject the boundaries
• 40:21of inquiry placed by proponents
• 40:23of AI,
• 40:24should we consider
• 40:25other issues such as the
• 40:27massive amounts of energy required
• 40:29to power AI?
• 40:31And this estimate here,
• 40:33says that data centers in
• 40:34the US,
• 40:36use somewhere around two hundred
• 40:38terawatt hours of electricity in
• 40:40twenty twenty four,
• 40:42which is roughly what it
• 40:43takes to power Thailand for
• 40:45a year.
• 40:47The electricity for these,
• 40:49data centers largely comes from
• 40:51fossil fuels because renewable energy
• 40:54is less reliable and not
• 40:55consistent enough
• 40:57to keep data,
• 40:59the amount,
• 41:00the minimum amount of energy
• 41:02required
• 41:03by data centers going.
• 41:05Millions of gallons of water
• 41:07are also used by data
• 41:08centers each day
• 41:10enough to deplete entire communities
• 41:12of their drinking water.
• 41:14So this is likely to
• 41:15have measurable, if unpredictable,
• 41:17health effects. So if we're
• 41:19thinking about the effects and
• 41:20the ethical issues raised by
• 41:22AI on health,
• 41:23we might wanna think broader
• 41:25than,
• 41:26bias and, privacy
• 41:28and fairness within the AI
• 41:30itself.
• 41:37So several years ago, Timnit
• 41:39Gebru identified bias in large
• 41:41language models that we now
• 41:43sort of just take for
• 41:44granted,
• 41:45but she got fired by
• 41:47her employer, Google, for her
• 41:48efforts.
• 41:50She then called out the
• 41:51colonization
• 41:52of poor and powerless nations
• 41:54by large AI companies
• 41:56and the exploitation of people
• 41:58who work for extremely low
• 41:59wages to label data that
• 42:01goes into the training models
• 42:03that we all use.
• 42:05And also the extraction of
• 42:07resources that do not belong
• 42:08to these companies,
• 42:10including data,
• 42:11intellectual property, as well as
• 42:13water and minerals such as
• 42:15copper and lithium.
• 42:17More recently, the author Karen
• 42:18Howe has also called out
• 42:20AI companies
• 42:21for this resource and power
• 42:22grab in her book called
• 42:24The Empire of AI.
• 42:26I suggest that we should
• 42:27not disengage from the broader
• 42:29moral questions raised by AI
• 42:31and not be confined to
• 42:33boundaries defined
• 42:34by technical considerations.
• 42:41There was a recent reprint
• 42:42that I just saw in
• 42:43my LinkedIn feed, yesterday.
• 42:46And it describes a comprehensive
• 42:48evaluation
• 42:49of the diagnostic and management
• 42:51reasoning capabilities
• 42:52of an advanced LLM with
• 42:54an OpenAI,
• 42:56product
• 42:57against a baseline of hundreds
• 42:59of physicians.
• 43:01I think they used the
• 43:02clinical reasoning cases that were
• 43:04developed, I as I hear,
• 43:05by doctor Duffy.
• 43:07They further studied the LLM's
• 43:10second opinions in a blinded
• 43:12fashion against expert physician baselines
• 43:14on randomly selected patients in
• 43:17a major academic tertiary care
• 43:19emergency department in, Boston.
• 43:22So the author suggests, as
• 43:23you can see in the
• 43:24title
• 43:25of their, preprint,
• 43:28that LLMs have achieved superhuman
• 43:30performance on general medical diagnostic
• 43:33and management reasoning.
• 43:36However, I suggest that reasoning
• 43:37alone is not a sufficient
• 43:39benchmark for evaluating patient care
• 43:41and the quality of it.
• 43:43I also suggest that these
• 43:45types of studies
• 43:46treat patients as data sources,
• 43:48as we saw in our
• 43:50in our study, and measure
• 43:51physician skills solely as data
• 43:54processors.
• 43:59So I submit that we
• 44:00should be less worried about
• 44:01whether AI will replace physicians
• 44:04and more concerned about
• 44:06whether patients will be replaced
• 44:08by data.
• 44:09This means looking at AI
• 44:10not only as a technology,
• 44:13but as part of the
• 44:14entire patient environment
• 44:15as a social determinant of
• 44:17health.
• 44:18And this was recently proposed
• 44:20by a postdoctoral fellow who
• 44:21works with me at Stanford,
• 44:22Nicole Foti.
• 44:24And,
• 44:25based on proposals that,
• 44:28modify,
• 44:29the familiar social determinants of
• 44:31health model, which was developed
• 44:33to address health disparities.
• 44:35And these modifications
• 44:36integrate digital determinants of health.
• 44:39These determinants include bias in
• 44:41data and algorithms that we
• 44:43are all familiar with, but
• 44:44also differential access to services
• 44:47and bandwidth on the Internet
• 44:49as well as uneven data
• 44:50protection and AI laws.
• 44:53Therefore, I will end here
• 44:55by urging us to consider
• 44:57whether we are morally disengaging
• 44:58from important ethical
• 45:00questions raised by AI in
• 45:02the medical context, and if
• 45:04so,
• 45:05to reengage
• 45:06as part of our professional
• 45:08duties.
• 45:09And in doing so, I
• 45:10hope that doctor Duffy would
• 45:11have agreed with me. Thank
• 45:13you for listening.
• 45:24I think we can sit
• 45:26here together.
• 45:43Thank you so much, doctor
• 45:44Cho.
• 45:47I I really appreciate your
• 45:49your your talk, and and
• 45:50I wanna take the,
• 45:53host's prerogative to to follow-up
• 45:55with with a question, and
• 45:56then I wanna open up
• 45:57to the to the audience
• 45:58for
• 45:59discussion and questions.
• 46:02You know, one thing I
• 46:04I found myself wondering as
• 46:05I was
• 46:06listening to your talk and
• 46:07listening to your,
• 46:09your very exciting
• 46:10work on on an intervention
• 46:12to
• 46:13to increase,
• 46:14ethical engagement and ethical action
• 46:18is that it
• 46:19as as we hear more
• 46:21about,
• 46:23AI or research AI development,
• 46:25it seems like there is
• 46:25enormous competitive pressure
• 46:28going on between
• 46:30the developers
• 46:31and also
• 46:33between countries.
• 46:34And that
• 46:36there's
• 46:37enormous pressure to
• 46:38put
• 46:39safeguards
• 46:40and
• 46:41ethical considerations to the side
• 46:43for developers, and there's also
• 46:45pressure
• 46:47to refrain
• 46:48from regulating
• 46:49on the part of nations
• 46:50because of fear that,
• 46:53other nations in competition are
• 46:55not going to regulate and
• 46:56that they'll lose,
• 46:58developers, they'll lose,
• 47:01advantages to other countries.
• 47:03It I'm I'm curious sort
• 47:05of,
• 47:06you know, as you're thinking
• 47:07about interventions to increase
• 47:10ethical engagement,
• 47:11how
• 47:12you
• 47:14how we might think about
• 47:15offsetting
• 47:16or addressing the this problem
• 47:19of the competitive pressures that
• 47:20developers and countries are under.
• 47:24Yeah. That's,
• 47:26something that is
• 47:27quite daunting.
• 47:30I think I would point
• 47:31to suggestions that Timnit Gebru
• 47:34had made and others,
• 47:36in in other countries as
• 47:38well,
• 47:40where,
• 47:42like minded developers
• 47:43who
• 47:44are interested
• 47:46in
• 47:47pursuing
• 47:49safe and effective AI for
• 47:51specific purposes
• 47:53joined together
• 47:54and,
• 47:57essentially,
• 47:58form their own groups and
• 47:59work on their own.
• 48:01So this has happened in
• 48:02New Zealand and in other
• 48:04countries,
• 48:05where people are collecting their
• 48:08own data
• 48:10and developing,
• 48:11their own AI tools
• 48:13where they are
• 48:16more likely to have
• 48:18gotten,
• 48:20input from local communities
• 48:22that these will serve their
• 48:23needs
• 48:24and also have buy in
• 48:26from the communities to produce
• 48:28data, to provide data
• 48:30for these efforts so that
• 48:31they will be more
• 48:33likely to actually bring benefits
• 48:36to the communities.
• 48:38And then
• 48:39also that,
• 48:42I think there's
• 48:44a growing emphasis or growing
• 48:45recognition
• 48:46that these general AI tools
• 48:49like these LLMs
• 48:50are,
• 48:52probably not appropriate to be
• 48:54used
• 48:54broadly for very specific medical
• 48:57applications.
• 48:58And so that you don't
• 48:59necessarily
• 49:00need to use
• 49:01these giant datasets,
• 49:03to get the effect that
• 49:05you're looking for,
• 49:07in an accurate and safe
• 49:08way.
• 49:12Let let me invite folks.
• 49:13If you'd like, we have
• 49:14a microphone set up,
• 49:15on this side of the
• 49:17auditorium,
• 49:18and, hopefully, we don't have
• 49:19one over there yet. But
• 49:20if folks would like to
• 49:21come up,
• 49:23to to the microphone,
• 49:31here.
• 49:38Thank you so
• 49:40much.
• 49:42One, two. Oh, there we
• 49:43go. Thank you so much.
• 49:44Long walk up.
• 49:47So I love this last,
• 49:49slide that you left us
• 49:50with, this last this concept
• 49:52of patient as data. And
• 49:54one of your colleagues at
• 49:55Stanford, Abraham Verghese,
• 49:57in two thousand eight wrote
• 49:58a very influential paper called
• 50:00patient as icon, icon as
• 50:02patient,
• 50:02where he introduced the concept
• 50:04of the iPatient.
• 50:06And I'm curious how you
• 50:07see this as different,
• 50:09patient as data,
• 50:10as patient from ICON.
• 50:14Well, can you say a
• 50:15little bit more about the
• 50:16this
• 50:18I'm not sure I heard
• 50:18you. Sure. Sorry.
• 50:20So Abraham Verghese at Stanford
• 50:23came up with this concept
• 50:24of the iPatient
• 50:25where he spoke about that,
• 50:28the
• 50:29electronic version or, of the
• 50:31patient
• 50:32is getting ex excellent care,
• 50:34but oftentimes, the actual patient
• 50:36at the bedside in their
• 50:37family doesn't know what's happening.
• 50:39And that the work of
• 50:40the physicians is oftentimes on
• 50:42the iPatient
• 50:43in this electronic version. And
• 50:44I'm curious about the concept
• 50:45of the iPatient and also
• 50:47now your concept of patient
• 50:48as data or data source
• 50:50if they're the same. Yeah.
• 50:51I think they're basically the
• 50:52same.
• 50:53People had started talking about
• 50:55this,
• 50:56many years ago when
• 50:59we started having, you know,
• 51:01things
• 51:03like wearables
• 51:04and and other things where,
• 51:07people be were being replaced
• 51:10by their data and,
• 51:12represented by their data and
• 51:14and sometimes misrepresented
• 51:16by their data. And, also
• 51:18so I think that
• 51:21maybe the thing that's that's
• 51:23more
• 51:23significant now is,
• 51:26to the extent that AI
• 51:28tools can bring all these
• 51:29data sources together.
• 51:31We can even make synthetic
• 51:33data, so we don't even
• 51:34need a person anymore. We
• 51:35can just make a synthetic
• 51:37person,
• 51:38that,
• 51:39that will be that will
• 51:41start to be seen as
• 51:43as we saw in this
• 51:44article that was published,
• 51:46that's the sort of benchmark
• 51:48that you're seeing as hitting
• 51:49is,
• 51:50treating the eye patient, not
• 51:53the
• 51:55actual
• 51:57patient.
• 51:59So thank you, doctor Shell.
• 52:01You know, one of the
• 52:02things that struck me particularly
• 52:04when you were giving the
• 52:05example of the,
• 52:08driverless car,
• 52:10that
• 52:11one of the questions that
• 52:12we have to ask is
• 52:14who is responsible
• 52:16for the helpfulness or harm
• 52:19of a technology. So, for
• 52:21example, if,
• 52:23if a driverless car
• 52:25hurt one person but was
• 52:27ultimately safer
• 52:29than the system that we
• 52:30have now on a on
• 52:31a larger scale.
• 52:34One could at least imagine
• 52:35an argument that that this
• 52:37is a technology that we
• 52:38would have to should pursue
• 52:40while accepting the consequences.
• 52:42And so
• 52:44shifting to the medical
• 52:46question, you know, a lot
• 52:47of what you talked about
• 52:48reminded me of almost the
• 52:49comparison between a public health
• 52:51approach
• 52:52to taking care of populations
• 52:55versus
• 52:55what we do as physicians
• 52:57taking care of individual patients.
• 52:59So
• 53:01so, for example, the the
• 53:02problem of the application of
• 53:03of
• 53:05a large dataset
• 53:07fails
• 53:07to the extent that it
• 53:08doesn't actually know the the
• 53:11details of the individual patients
• 53:13sitting with you
• 53:14in the exam room. So
• 53:15so getting back to to
• 53:17the moral aspect of this,
• 53:20how do you how would
• 53:21you approach
• 53:22labeling the the good or
• 53:24the bad or the helpfulness
• 53:25or the harm
• 53:27of
• 53:28these AI tools
• 53:31when you realize it may
• 53:32help a lot of people
• 53:33but harm individuals,
• 53:36which is already going on
• 53:37pre AI, but but perhaps
• 53:39a way of thinking about
• 53:41how to judge the value
• 53:42or or lack of value
• 53:43of AI tools.
• 53:46Yeah. Well, you had to
• 53:48ask the easy question. I
• 53:50think I need help from
• 53:51my bioethics colleagues over here.
• 53:53But, I mean, I think
• 53:54we have faced this question
• 53:56a lot. Right? I mean
• 53:57so there's always this tension
• 53:59between the individual and the
• 54:01population.
• 54:03And one way that we
• 54:05look at that is if
• 54:06we're looking at population effects
• 54:08or population
• 54:10interventions,
• 54:11we tend to set a
• 54:12very, very high standard
• 54:15for those. Right? We don't
• 54:16use them in all cases.
• 54:18There have to be
• 54:19clear,
• 54:21clear benefits
• 54:22from using it at the
• 54:24population level,
• 54:26and,
• 54:27high standards for efficacy.
• 54:30So that's one thing that
• 54:31I think is not happening
• 54:32right now.
• 54:33And then in terms of
• 54:34weighing the many versus the
• 54:36one, you know, I think
• 54:37that you can't really
• 54:39I mean, I know I
• 54:40talk with my students about
• 54:41this idea of, effective altruism
• 54:44and these,
• 54:45kinds of extreme
• 54:47extreme arguments,
• 54:49about, you know,
• 54:51utilitarianism.
• 54:52So I think we
• 54:53we need to apply those
• 54:54same
• 54:55principles where we don't go
• 54:57to the extremes of utilitarianism,
• 55:00for many reasons.
• 55:03So and in addition, I
• 55:04think,
• 55:06I think what you're what
• 55:07you said
• 55:09also applies, which is that
• 55:11right now, AI tools or
• 55:13even the whole concept of
• 55:14precision medicine,
• 55:16which is supposed to take
• 55:18aggregate data and apply it
• 55:19to individuals still has a
• 55:21gap.
• 55:22Right? We can't do that,
• 55:23and that's what happens in
• 55:24a physician's head.
• 55:26Right? They're trying to bridge
• 55:27that gap between the aggregate
• 55:29data
• 55:30on how to treat patients
• 55:32that are like this patient
• 55:33and then how to treat
• 55:34this actual patient that's sitting
• 55:36in front of you.
• 55:37And and the AI has
• 55:38not bridged that gap yet.
• 55:41Thank you.
• 55:44Hi. Thank you for the
• 55:45talk.
• 55:46I'm an undergraduate student studying
• 55:48cognitive science and AI, so
• 55:49this was really in the
• 55:50niche that I'm very interested
• 55:51in, so thank you.
• 55:53I had a few questions,
• 55:54but I'll narrow it down
• 55:55into something that you talked
• 55:57more about, which is kind
• 55:58of this perception of having
• 55:59not a patient, but, like,
• 56:01their their digital information
• 56:03with, a bit of a
• 56:04negative connotation,
• 56:06which I wanted to know
• 56:07in the aspect of trying
• 56:09to remove racial gender
• 56:11bias that are kind of
• 56:12inherent and something that's very
• 56:13prevalent in everyone's cognition, whether
• 56:15they like it or not,
• 56:17how do you see the
• 56:17importance of
• 56:19pure data with wearables,
• 56:22kind of both emerging,
• 56:24the
• 56:25taking maybe a little bit
• 56:26of a devil's advocate
• 56:28view of the research that
• 56:29you did give, what are
• 56:30the benefits that you could
• 56:32imagine seeing aside from the
• 56:33few that I mentioned?
• 56:35Okay. Great question.
• 56:38I think this is where
• 56:40this issue of the digital
• 56:42determinants of health come in,
• 56:45which is that
• 56:47only
• 56:48very wealthy privileged people get
• 56:50to have all this data
• 56:51taken about them, and that
• 56:53shapes all the AI, you
• 56:55know, algorithms that we know.
• 56:58So there's deep biases built
• 56:59into the data collection
• 57:01and also to the application
• 57:03of it afterwards.
• 57:04So,
• 57:06whereas it could be I
• 57:08mean, people make the argument
• 57:09that AI will be less
• 57:11biased than humans.
• 57:13But,
• 57:14right now, we're sort of
• 57:15in a vicious cycle where
• 57:17we can't really get out
• 57:18of our we can't get
• 57:18of our own out of
• 57:19our own way.
• 57:22So,
• 57:23I think that, you know,
• 57:26this just goes to this
• 57:27is gonna I just hope
• 57:28that this is not another
• 57:29example of technologies using to
• 57:32exact being used to exacerbate
• 57:34existing
• 57:35disparities.
• 57:36So people who have less
• 57:38data about them
• 57:39will be less able to
• 57:41benefit from the data that's
• 57:43out there provided by others.
• 57:44And do you need health
• 57:45data,
• 57:46specifically for health? Because if
• 57:48you're thinking about if someone's
• 57:49using their iPhone and all
• 57:50of the data that's being
• 57:51collected with the cookies and
• 57:53kind of targeted advertisements,
• 57:54that's has quite a large
• 57:56equilibrium for, like, everyone depending
• 57:58on race and, like, the
• 58:00specific accuracy of advertisement has
• 58:02proven that we do have
• 58:03a lot of data on
• 58:04people. We do know what
• 58:04they're interested in, especially in
• 58:05a health situation. Do you
• 58:07mean it more for wearables
• 58:08and, you know, products that
• 58:10people have to buy that
• 58:11are monitoring, like, their heart
• 58:12rate or other physical. Well,
• 58:14I just think that think
• 58:16about who has wearables and
• 58:17who does not. Right. Who
• 58:18has iPhones, who does not.
• 58:20Who has Internet, who does
• 58:21not. I live in a
• 58:23neighborhood that is surrounded by
• 58:25Silicon Valley billionaires,
• 58:26and I don't have cell
• 58:28phone service at my house
• 58:29because it's in a rural
• 58:30area.
• 58:31So I think about my
• 58:33neighbors who are not Silicon
• 58:34Valley
• 58:35billionaires,
• 58:36and none of us can
• 58:38use Uber.
• 58:39None of us can use
• 58:40DoorDash.
• 58:41And none of us can,
• 58:43get
• 58:45access to telehealth
• 58:47applications. So there's huge disparities
• 58:49here that I think are
• 58:51ignored by people who are
• 58:55proponents of these technologies that
• 58:57require
• 58:58a lot of resources
• 59:00to be able to access.
• 59:02I see. Thank you.
• 59:06Hi. Thanks so much for,
• 59:07for the talk and the
• 59:08framing.
• 59:09As I think about the,
• 59:11clinical application
• 59:13of these AI tools,
• 59:15one pattern that has emerged,
• 59:17especially with the very,
• 59:18I would say, loose regulatory
• 59:20framework is a lot of
• 59:21nondevice clinical decision support.
• 59:24And I think
• 59:25as we think about this
• 59:26idea of the copilot, you
• 59:28mentioned this with the moral
• 59:29disengagement. Right? I'm not telling
• 59:31the doctor what to do.
• 59:32The copilot is just suggesting
• 59:34things, and then the doctor
• 59:35makes the decision.
• 59:37I think that framework
• 59:39is vaguely justifiable
• 59:41when the algorithms are really
• 59:44serving more as
• 59:45a deterministic
• 59:47representations
• 59:48of possible diagnostic
• 59:50possibilities. Right? So could be
• 59:51one of these six think
• 59:52about these ten things and
• 59:54you choose.
• 59:55But as the models get
• 59:56more sophisticated
• 59:57and the explainability
• 59:59starts to disappear,
• 01:00:00so these models are more
• 01:00:01about discovery than they are
• 01:00:03about confirmation.
• 01:00:04You have a digital pathology
• 01:00:06algorithm that says,
• 01:00:07this is a breast cancer,
• 01:00:09but the pathologist can't tell
• 01:00:11that. They they they're not
• 01:00:12able to see what the
• 01:00:13model can see.
• 01:00:15I think increasingly with this
• 01:00:17idea that the model is
• 01:00:18just providing advice starts to
• 01:00:21become very hard to defend.
• 01:00:23And so I wonder what
• 01:00:24your perspective is about
• 01:00:25sort of the moral dilemma
• 01:00:27that the physician
• 01:00:29or the provider is then
• 01:00:30placed in when they don't
• 01:00:32feel competent to evaluate
• 01:00:34the the model's,
• 01:00:36outcome prediction. And what is
• 01:00:38their obligation in that scenario?
• 01:00:40Is it for the in
• 01:00:41the patient's best interest? Is
• 01:00:42it to defer to the
• 01:00:43model? Because
• 01:00:44in the performance analysis, the
• 01:00:46model is superhuman. Is it
• 01:00:47to override the model because
• 01:00:49of its lack of explainability?
• 01:00:50Is it to say, I
• 01:00:51won't use a model
• 01:00:52that is not readily explainable?
• 01:00:54How how do you think
• 01:00:55about
• 01:00:56the the moral obligation of
• 01:00:58the physician in that experience
• 01:00:59when
• 01:01:00the model is now being
• 01:01:01made available to practitioners?
• 01:01:04Yeah.
• 01:01:06Again, a super simple question.
• 01:01:08Mhmm.
• 01:01:10I mean, when I raise
• 01:01:12this question with other clinicians,
• 01:01:16many say,
• 01:01:17well, I'm in that position
• 01:01:18all the time where I
• 01:01:19don't know what's going on.
• 01:01:21I don't know why. There's
• 01:01:22never any explanations for any
• 01:01:24of the lab results or
• 01:01:25anything like that. So why
• 01:01:26should this be any different?
• 01:01:29So on the one hand,
• 01:01:32maybe,
• 01:01:33but I think
• 01:01:34I think what's different here
• 01:01:37is that
• 01:01:39there is
• 01:01:40there's no evaluative
• 01:01:42standards
• 01:01:43that are put into place
• 01:01:44if a system like that,
• 01:01:45like a diagnostic thing in
• 01:01:47pathology gets installed
• 01:01:49in a health system?
• 01:01:50How is their general assurance?
• 01:01:53And part of the problem
• 01:01:55is that we've seen with
• 01:01:56Epic and other, like the
• 01:01:58Epic sepsis system and all
• 01:02:00that, that, you know, they
• 01:02:02seem like they perform great
• 01:02:03when they,
• 01:02:04when they are first reported
• 01:02:06on whatever dataset that was
• 01:02:08used to generate it, and
• 01:02:09we don't know what the
• 01:02:10datasets are because they're not
• 01:02:12transparent about that. And then
• 01:02:13it gets implemented
• 01:02:14in a different situation, so
• 01:02:16you actually don't have an
• 01:02:17evaluation
• 01:02:18of the performance in your
• 01:02:20setting.
• 01:02:21So I think that clinicians
• 01:02:22could demand this
• 01:02:24and say that they won't
• 01:02:25use something unless it has
• 01:02:26been evaluated
• 01:02:27in their setting, in their
• 01:02:28health care system,
• 01:02:31because they don't have assurances
• 01:02:32that someone's got their back
• 01:02:34on these on these,
• 01:02:36diagnoses or decisions that are
• 01:02:38being made. Yeah. I think
• 01:02:39that,
• 01:02:40that's a general principle. I
• 01:02:42think that many of us
• 01:02:43would apply when we're taking
• 01:02:45when we're trying to do
• 01:02:46a silent validation of an
• 01:02:47algorithm
• 01:02:48for which the answer is
• 01:02:49readily apparent. But some of
• 01:02:50these algorithms are your risk
• 01:02:52of x in five years.
• 01:02:54And so it becomes,
• 01:02:56just technically unfeasible
• 01:02:59to wait five years before
• 01:03:00deciding whether or not to
• 01:03:02release a tool like that.
• 01:03:03So
• 01:03:04I think it's just a
• 01:03:05I think it's gonna be
• 01:03:06an increasingly,
• 01:03:07challenging moral dilemma for physicians.
• 01:03:09Do do I override the
• 01:03:11model
• 01:03:12and face the risk of
• 01:03:13harming my patient because the
• 01:03:15model actually knows better? Do
• 01:03:17I adhere you know, defer
• 01:03:19to the model? And how
• 01:03:20do I create a system
• 01:03:22of accountability
• 01:03:23to ensure that as I
• 01:03:24follow that patient over time,
• 01:03:26I can record
• 01:03:28whether or not my patient
• 01:03:29benefited from that experience and
• 01:03:31have that aggregated at some
• 01:03:32level. So thank you. I
• 01:03:33mean, I think that sort
• 01:03:34of assumes this sense of
• 01:03:36inevitability
• 01:03:37that we must use the
• 01:03:38model because it's new and
• 01:03:39it came out, and there
• 01:03:40it is.
• 01:03:42I mean, other people have
• 01:03:44I mean, I understand this
• 01:03:45this dilemma of having to
• 01:03:47wait five years to find
• 01:03:48out whether it's right. But,
• 01:03:50I mean, we don't do
• 01:03:51that with other kinds of
• 01:03:52health interventions.
• 01:03:53We we wait the five
• 01:03:54years. Right? I mean,
• 01:03:57or
• 01:03:59maybe there's ways of running
• 01:04:01it in parallel as a
• 01:04:02shadow system with other things
• 01:04:04that have more transparency,
• 01:04:07while you wait and evaluate.
• 01:04:10I think it's it may
• 01:04:12be a I I I
• 01:04:13am I get the feeling
• 01:04:14that,
• 01:04:17I get uneasy with this
• 01:04:19sense of inevitability.
• 01:04:22Thank you.
• 01:04:27I think it was, the
• 01:04:28senate late senator,
• 01:04:30Everett Dirksen, who said,
• 01:04:33a billion here, a billion
• 01:04:35dollars there. Pretty soon, you're
• 01:04:36talking real money.
• 01:04:39When you alluded to,
• 01:04:42some of the disparities
• 01:04:43that are occurring
• 01:04:45because of AI, you alluded
• 01:04:47to energy
• 01:04:49and water.
• 01:04:51And I
• 01:04:53had only recently begun to
• 01:04:54use an AI program,
• 01:04:57to answer some questions in
• 01:04:59medicine, and and I've only
• 01:05:00recently become aware
• 01:05:02that I'm inserting myself into
• 01:05:05a system
• 01:05:06that is consuming
• 01:05:07a huge amount of energy.
• 01:05:10And that the effects
• 01:05:12of
• 01:05:13AI
• 01:05:14are likely to be equivalent
• 01:05:16to the invention of the
• 01:05:17internal combustion engine on the
• 01:05:19consumption of energy, only it'll
• 01:05:21occur much, much more rapidly.
• 01:05:24It is occurring very rapidly.
• 01:05:26I would like to hear
• 01:05:27your thoughts about that disparity
• 01:05:30because
• 01:05:32when energy has to go
• 01:05:33into
• 01:05:34this,
• 01:05:35it's gonna be taken away
• 01:05:37from someone else
• 01:05:38because there isn't enough energy
• 01:05:40to go around at the
• 01:05:41rate we're going. At least,
• 01:05:42I don't think there is.
• 01:05:45Well yeah. So I'm not
• 01:05:47an expert on, you know,
• 01:05:49power systems, but I do
• 01:05:51think that
• 01:05:57it's something that
• 01:06:00we should all consider,
• 01:06:03because people can vote with
• 01:06:05their feet and not use
• 01:06:07technologies
• 01:06:08that they feel are
• 01:06:10detrimental
• 01:06:12because of perhaps overuse of
• 01:06:14power and water.
• 01:06:17Different
• 01:06:20communities
• 01:06:21who are faced with, thanks,
• 01:06:23who are faced with
• 01:06:26loss of their water because
• 01:06:27of data centers moving in
• 01:06:29are are able to block
• 01:06:31the building of data centers
• 01:06:33in their in their neighborhoods,
• 01:06:35because of
• 01:06:38sorry.
• 01:06:41Thanks.
• 01:06:44So
• 01:06:45it may just be that
• 01:06:46this is something that can
• 01:06:47only be dealt with at
• 01:06:48the political level
• 01:06:50because it's so big. I
• 01:06:51mean, it's really
• 01:06:52out of the hands of
• 01:06:54of
• 01:06:55individual physicians or even health
• 01:06:57systems. These are these are
• 01:06:59much bigger questions
• 01:07:01about how to use resources.
• 01:07:02And
• 01:07:05Yes. Thank you.
• 01:07:11Thank you, doctor Cho, for
• 01:07:13your
• 01:07:14phenomenal presentation,
• 01:07:17in an area on the
• 01:07:18subject that we really need
• 01:07:19to focus on.
• 01:07:21But my question got to
• 01:07:23do with something much more
• 01:07:24fundamental.
• 01:07:27Quite some time ago,
• 01:07:29our students,
• 01:07:31particularly
• 01:07:33our which is our
• 01:07:34being Yale School of Medicine,
• 01:07:37they are our priceless possession.
• 01:07:40They gave up reading textbooks.
• 01:07:47The latest version of Harrison
• 01:07:49is on my iPhone,
• 01:07:52Kindle app, etcetera,
• 01:07:54etcetera.
• 01:07:55But they don't read it
• 01:07:56read textbooks anymore.
• 01:07:59We caught on to that
• 01:08:00a little late.
• 01:08:02They were reading all kinds
• 01:08:04of things and,
• 01:08:06mostly online
• 01:08:08and
• 01:08:09some,
• 01:08:11on paper.
• 01:08:13And I think we
• 01:08:16really, therefore, didn't know where
• 01:08:18they were going with
• 01:08:21it, whether they were in
• 01:08:22the right direction
• 01:08:23or not.
• 01:08:26I think it's something
• 01:08:28urgent
• 01:08:29that we need to focus
• 01:08:30on
• 01:08:31to introduce even while we
• 01:08:33are still struggling with the
• 01:08:35methodology
• 01:08:35of developing teaching tools, etcetera,
• 01:08:38etcetera,
• 01:08:40our students are already reusing
• 01:08:42AI
• 01:08:43in in different ways.
• 01:08:46That how do you
• 01:08:48suggest
• 01:08:50in the curriculum,
• 01:08:52the preclinical
• 01:08:53student curriculum
• 01:08:55at that level,
• 01:08:58formally introducing
• 01:08:59and directing our students
• 01:09:02in the use of
• 01:09:04AI
• 01:09:05when they should,
• 01:09:07where they should go, for
• 01:09:08what kind of information,
• 01:09:11what they should do once
• 01:09:12they become clinical students. Unlike
• 01:09:14you and me, they didn't
• 01:09:15spend two years in preclinical.
• 01:09:17They spent only a year
• 01:09:18and a half now
• 01:09:20before they move on.
• 01:09:22So that's what I seek.
• 01:09:25I think we all seek
• 01:09:26your advice for. At what
• 01:09:28stage, how,
• 01:09:30and in what fashion should
• 01:09:31that curriculum be designed? Because
• 01:09:33we missed the bus once.
• 01:09:36We can't miss this one.
• 01:09:38Thank you.
• 01:09:41Well,
• 01:09:41I
• 01:09:42can't give you an answer
• 01:09:43to that question. I think
• 01:09:44it's something that all educators
• 01:09:46are struggling with. But,
• 01:09:50you know, there's lots of
• 01:09:51different kinds of AI tools
• 01:09:53out there. So, you know,
• 01:09:55there's going to have to
• 01:09:56be a lot of different
• 01:09:57kinds of recommendations
• 01:09:58based on the different kinds
• 01:10:00of tools. And these large
• 01:10:01language models that we're looking
• 01:10:03at
• 01:10:04are probably
• 01:10:06really only good for
• 01:10:07a very small number of
• 01:10:09things,
• 01:10:10really, even though they're posed
• 01:10:11as general tools. They're
• 01:10:14as as I think of
• 01:10:15them, they're basically,
• 01:10:17like autofill on steroids,
• 01:10:21which isn't
• 01:10:22necessarily,
• 01:10:24you know, just predicting the
• 01:10:25next word based on how,
• 01:10:27you know, sentences have occurred
• 01:10:29in the past is not
• 01:10:30really,
• 01:10:31very useful for a lot
• 01:10:33of different things that students
• 01:10:34need to do, and it's
• 01:10:35certainly not very good for
• 01:10:36critical thinking skills,
• 01:10:39development. So,
• 01:10:41I mean, I've been struggling
• 01:10:42with this question myself for
• 01:10:44my own teaching of, an
• 01:10:46ethics course.
• 01:10:47And
• 01:10:48one of the ideas that
• 01:10:49I've had is to
• 01:10:52have students
• 01:10:54develop prompts to ask CHAT
• 01:10:56GPT
• 01:10:57certain things,
• 01:10:59and then critique the responses
• 01:11:01so that I hope that
• 01:11:03they learn from independent sources
• 01:11:05about the topic that they're
• 01:11:06asking chat g p t.
• 01:11:09But I think it's a
• 01:11:11very
• 01:11:11difficult question. It's also
• 01:11:14necessary to train people to
• 01:11:16understand
• 01:11:18when they're looking at something
• 01:11:19fake and something real because
• 01:11:21that's increasingly difficult to tell
• 01:11:23the difference between.
• 01:11:25And so
• 01:11:27I think that's another
• 01:11:28very
• 01:11:30difficult and disturbing task that
• 01:11:33we now have to face,
• 01:11:34that we didn't have to
• 01:11:35face before.
• 01:11:36But just finally, on,
• 01:11:39on the other side, I
• 01:11:40think that
• 01:11:41some of the trainees that
• 01:11:43I work with,
• 01:11:44have very quickly picked up
• 01:11:46on,
• 01:11:48prompt engineering skills, which I
• 01:11:50do not have myself.
• 01:11:52And,
• 01:11:53you can get much better
• 01:11:55results,
• 01:11:56in certain kinds of tasks,
• 01:11:58like writing tasks,
• 01:12:00if you are very good
• 01:12:01at writing very good prompts.
• 01:12:04Again, I'm not one of
• 01:12:06those people. But,
• 01:12:09you know, I think it
• 01:12:10means teaching
• 01:12:11teaching students or allowing people
• 01:12:13to learn not by themselves,
• 01:12:16what the limitations of these
• 01:12:18kinds of AI tools are.
• 01:12:24Can
• 01:12:26I just interject and and
• 01:12:28ask in in our
• 01:12:30professional responsibility
• 01:12:31course,
• 01:12:32led by Dave Rosenthal and
• 01:12:34Karen Jovanik, who I think
• 01:12:36were, in the audience,
• 01:12:39Within our small group, we
• 01:12:41we had a a vote
• 01:12:42about whether to exclude devices
• 01:12:45from the the, small group
• 01:12:46classroom and the and the
• 01:12:48students overwhelmingly voted in favor
• 01:12:49of removing devices? And I
• 01:12:50think it's it's been a,
• 01:12:53positive experience thus far. This
• 01:12:54is the first year that
• 01:12:55we we had this this
• 01:12:56kind
• 01:12:57of consideration.
• 01:12:59I'm I'm curious,
• 01:13:00you know, on the on
• 01:13:01the other hand,
• 01:13:02we're not engaging with with
• 01:13:04with AI. We're we're, you
• 01:13:05know, intentionally
• 01:13:07having a a very,
• 01:13:09low tech,
• 01:13:10experience.
• 01:13:12What are your thoughts about
• 01:13:13that? On the one hand,
• 01:13:16AI has no role in
• 01:13:17in the classroom. On the
• 01:13:18other hand, we're not engaging
• 01:13:21with
• 01:13:22critiques of
• 01:13:24the AI output. We're not
• 01:13:26engaging with prompt engineering. What
• 01:13:28are your thoughts about balancing
• 01:13:30those?
• 01:13:31Well,
• 01:13:41I you know, given that
• 01:13:42it's a professional responsibility
• 01:13:44course, I would say that
• 01:13:49I
• 01:13:50it may not be necessary
• 01:13:52to engage with AI,
• 01:13:53or if you do, only
• 01:13:55to the extent that it
• 01:13:56is part of professional responsibility.
• 01:13:58So if you
• 01:13:59you could argue that it
• 01:14:01is now a physician's responsibility
• 01:14:04to
• 01:14:05understand
• 01:14:06limitations of AI, understand how
• 01:14:09it
• 01:14:10helps or hinders
• 01:14:11communication with patients, how it
• 01:14:13helps or hinders, you know,
• 01:14:14what are their own responsibilities
• 01:14:16in using AI in the
• 01:14:18health care system.
• 01:14:19So maybe that's
• 01:14:21one question, but you don't
• 01:14:23necessarily need to use AI
• 01:14:24tools or devices in the
• 01:14:26course to achieve those goals.
• 01:14:32And that segues, I think,
• 01:14:34to a part of my
• 01:14:35question, which is to get,
• 01:14:38to your point about engaging
• 01:14:39bioethicists
• 01:14:40in this and this idea
• 01:14:42of professional responsibility and the
• 01:14:44role of AI in that
• 01:14:45identity. So I am coming
• 01:14:47from a law school,
• 01:14:48and today's lunch was a
• 01:14:50conversation of what is, if
• 01:14:51there is, the ethical use
• 01:14:53of AI in legal scholarship
• 01:14:55and legal research. And so
• 01:14:57we have a whole bunch
• 01:14:58of researchers in this room.
• 01:15:00We have a whole bunch
• 01:15:01of people who want to
• 01:15:02present
• 01:15:04ideally
• 01:15:05original ideas,
• 01:15:06but to the extent that
• 01:15:09increased and improved
• 01:15:10prompt engineering
• 01:15:13can easily help and write
• 01:15:15things, revise things, reorganize things,
• 01:15:18essentially do some virtual workshopping
• 01:15:21of papers.
• 01:15:22Do you have a sense
• 01:15:24of
• 01:15:25specifically in bioethics,
• 01:15:27whether or not we need
• 01:15:28to be having a conversation
• 01:15:30of the appropriate use of
• 01:15:31AI
• 01:15:32in generating the articles we
• 01:15:35seek to place and review
• 01:15:37and accept?
• 01:15:39Yeah. Another hot topic of
• 01:15:42discussion.
• 01:15:44I know that the,
• 01:15:46committee of on publication,
• 01:15:49ethics, COPE, or you don't
• 01:15:50know if you're familiar with
• 01:15:51this group,
• 01:15:53came out recently with
• 01:15:55a,
• 01:15:57a guideline
• 01:15:58that
• 01:16:00says that they don't consider
• 01:16:02AI to be an author.
• 01:16:03And they would suggest that
• 01:16:05journals make a statement that
• 01:16:07the their member journals make
• 01:16:09a statement that that AI
• 01:16:11cannot be an author.
• 01:16:13And I think they have
• 01:16:14good reasons for that. And
• 01:16:15that is that, at least
• 01:16:17as we teach it in
• 01:16:19our responsible conduct of research
• 01:16:21course,
• 01:16:22being an author means that
• 01:16:23you are accountable,
• 01:16:25not just for the words
• 01:16:26that are written, but for
• 01:16:28everything that went into it
• 01:16:29and the data
• 01:16:31collection and all that stuff.
• 01:16:32Right? I mean, that's what
• 01:16:34scientific responsibility
• 01:16:37is. It's it's all these
• 01:16:38things, not just the written
• 01:16:39words.
• 01:16:40And,
• 01:16:42a chatbot cannot be accountable
• 01:16:44for anything, and it's especially
• 01:16:46cannot be accountable for all
• 01:16:47the things that
• 01:16:48that happened leading up to
• 01:16:50the writing. Right? So the
• 01:16:52writing is not just
• 01:16:55it's a representation
• 01:16:58of a lot of work
• 01:16:59and thought, not just the
• 01:17:01words themselves. So
• 01:17:03I think
• 01:17:05we have to think about
• 01:17:08things like publications
• 01:17:09as as just the surface
• 01:17:12and that there has to
• 01:17:13be an accountable body underneath
• 01:17:15it.
• 01:17:20Thank you, Mildred, for the
• 01:17:22great talk.
• 01:17:27I wanna zoom in on
• 01:17:28something that you said almost
• 01:17:30in passing.
• 01:17:31You said,
• 01:17:32of course, we didn't use
• 01:17:34the word ethics because we
• 01:17:35didn't want them to become
• 01:17:36defensive.
• 01:17:39And that really caught my
• 01:17:40attention because
• 01:17:41bioethicists
• 01:17:42like us are now, because
• 01:17:44of the acute interest in
• 01:17:45this topic,
• 01:17:47going around the country, sitting
• 01:17:48on all the panels with
• 01:17:50developers and CEOs and clinicians
• 01:17:52and patients. And
• 01:17:54as soon as I am
• 01:17:55on the stage wearing the
• 01:17:56bioethicist
• 01:17:57hat,
• 01:17:58the expectation
• 01:18:00is that I will be
• 01:18:01the ethics police,
• 01:18:03that I will be the
• 01:18:04voice that says, why not,
• 01:18:06why not yet,
• 01:18:08concerns, issues, challenges, risks, dangers.
• 01:18:12I will be the naysayer
• 01:18:14in the context
• 01:18:15of mostly
• 01:18:17enthusiasts.
• 01:18:19And I resist this labeling.
• 01:18:23First of all,
• 01:18:25the grounds of how I
• 01:18:26see our field, I think
• 01:18:28that often it's our job
• 01:18:30to be facilitators,
• 01:18:32the opposite of naysayers.
• 01:18:34Because once we engage with
• 01:18:36innovation,
• 01:18:38and call for responsible
• 01:18:40wise use,
• 01:18:42our impact will be much
• 01:18:43more important,
• 01:18:44and we will find ourselves
• 01:18:45in the correct rooms around
• 01:18:46the correct tables.
• 01:18:49So I'm going back to
• 01:18:50what you said about I'm
• 01:18:51not comfortable with the inevitability
• 01:18:54of it.
• 01:18:55And at the Hastings Center
• 01:18:56for Bioethics, Virginia is here.
• 01:18:58She could testify. We always
• 01:19:00try to distinguish between questions
• 01:19:02of should we
• 01:19:04do it and questions of
• 01:19:06how should we do it
• 01:19:07and when we transition from
• 01:19:08one to the other.
• 01:19:10And my problem with this,
• 01:19:13approach of I don't I'm
• 01:19:15not comfortable with the inevitability
• 01:19:16is that we all know
• 01:19:17what's happening now.
• 01:19:19We all know it is
• 01:19:21inevitable.
• 01:19:22So in that context,
• 01:19:24first of all, the reality
• 01:19:26is inevitable. And second of
• 01:19:27all, that as a field,
• 01:19:29we will probably be much
• 01:19:31more effective as,
• 01:19:33yay. We're not the naysayers.
• 01:19:35We're the facilitators, but we're
• 01:19:37here to tell you
• 01:19:38how you can mitigate the
• 01:19:40risks, how you can make
• 01:19:41wiser choices as you try
• 01:19:43to teach to to sensitize
• 01:19:44your developers.
• 01:19:48What is our role
• 01:19:49now in this context? How
• 01:19:51to be
• 01:19:52around the right tables and
• 01:19:53still
• 01:19:54wear the emphasis hat,
• 01:19:57but not be perceived
• 01:19:59as we often are?
• 01:20:03Well,
• 01:20:05I think
• 01:20:06at some level,
• 01:20:10bioethicists
• 01:20:11have already been run over
• 01:20:12by the AI train.
• 01:20:14I mean, these are trillions
• 01:20:15of dollars at stake.
• 01:20:17We're the fly on the
• 01:20:19windscreen, and we've been smashed.
• 01:20:20We're just, you know,
• 01:20:22we're
• 01:20:23roadkill on that in that
• 01:20:25scenario.
• 01:20:26I don't think that we
• 01:20:28are going to have much
• 01:20:30of an impact
• 01:20:34as
• 01:20:35facilitators
• 01:20:36in modulating
• 01:20:38how the train is going.
• 01:20:40It's just going at warp
• 01:20:42speed, and it is fueled
• 01:20:44by
• 01:20:45a lot of money and
• 01:20:46a lot of interest.
• 01:20:48I just don't think
• 01:20:54it's something that we have
• 01:20:55much power to do anything
• 01:20:57about,
• 01:20:59except maybe at some very
• 01:21:03micro levels. I mean,
• 01:21:06I did not think that
• 01:21:08five years ago when we
• 01:21:09started the studies we were
• 01:21:10doing, they I I'm I'm
• 01:21:11looking back at the studies
• 01:21:13that we started all those
• 01:21:14years ago, and they seem
• 01:21:15very naive now.
• 01:21:17I'm good. That said, I'm
• 01:21:19going to still try to,
• 01:21:21intervene probably with people at
• 01:21:23the training stages of AI
• 01:21:25to have these discussions with
• 01:21:27them,
• 01:21:28to try to be more
• 01:21:29reflexive about what they're doing.
• 01:21:31But,
• 01:21:32as you've seen,
• 01:21:33even people who are inside
• 01:21:35these companies that are
• 01:21:39developing AI, as soon as
• 01:21:41they start getting in the
• 01:21:42way, they get fired
• 01:21:44Whole departments of ethics,
• 01:21:47AI ethics have been
• 01:21:50disentang disdemolished.
• 01:21:53So
• 01:21:55I
• 01:21:57I'm not sure that answers
• 01:21:59your question, but I think
• 01:22:00that it's
• 01:22:02to think that
• 01:22:06bioethicists
• 01:22:08are part of this AI
• 01:22:11program
• 01:22:12is,
• 01:22:14I mean, I wish that
• 01:22:15it were true.
• 01:22:22Hi. Thank you so much
• 01:22:23for a wonderful talk.
• 01:22:25I think when I think
• 01:22:26of the ethics of this
• 01:22:27issue more generally, I think
• 01:22:29about also access to care
• 01:22:31and the physician shortage
• 01:22:33and the thought process that
• 01:22:35potentially
• 01:22:36AI can be harnessed to
• 01:22:37help mitigate some of the
• 01:22:38effects of the physician shortage,
• 01:22:40particularly
• 01:22:41in rural areas or other
• 01:22:42places that are lacking access
• 01:22:44to care in our country
• 01:22:45and internationally,
• 01:22:47and how we can actually
• 01:22:48kind of turn the idea
• 01:22:49of the ethics of AI
• 01:22:51into more of a positive
• 01:22:52connotation as well in terms
• 01:22:54of what can we do
• 01:22:55to facilitate,
• 01:22:56kind of similar to what
• 01:22:57you were saying,
• 01:22:59the use of AI in
• 01:23:00some of these underrepresented
• 01:23:02populations or, you know, places
• 01:23:04that lack access
• 01:23:05so we can kind of
• 01:23:06help filter the most vulnerable
• 01:23:08people into
• 01:23:09care models that they need
• 01:23:10to be in.
• 01:23:12Yeah. I mean, I completely
• 01:23:14agree. And if you're one
• 01:23:15of those people that can
• 01:23:16help make that happen, then
• 01:23:18you, you know, you should.
• 01:23:19I I don't mean to
• 01:23:20be all doom and gloom,
• 01:23:22in in spite of what
• 01:23:23I said to Margit.
• 01:23:24But,
• 01:23:26I think that means that,
• 01:23:28there has to be a
• 01:23:28lot of attention
• 01:23:31to actually making these things
• 01:23:32happen. And the incentives
• 01:23:34the business incentives,
• 01:23:36to do that
• 01:23:38are
• 01:23:39are something that have to
• 01:23:40be
• 01:23:42surmounted.
• 01:23:43They're they're not running in
• 01:23:45that direction.
• 01:23:46So,
• 01:23:48you know, we have to
• 01:23:49be very conscious and fight
• 01:23:51hard
• 01:23:52to to ensure that this
• 01:23:54technology is beneficial and does
• 01:23:56not create greater
• 01:23:58disparities
• 01:23:59where they already exist.
• 01:24:04I have a different thought
• 01:24:06about
• 01:24:07AI. It's out of the
• 01:24:08box. It's the genie has
• 01:24:10opened up and it's out
• 01:24:11there. It's ahead of us.
• 01:24:14I wonder
• 01:24:15if an alternative way of
• 01:24:17thinking about it is us
• 01:24:19as consumers of health and
• 01:24:20health care.
• 01:24:22Right? If there's any chance
• 01:24:24of fighting the extreme amount
• 01:24:26of money, it's unclear
• 01:24:28what AI's benefit is as
• 01:24:30a business model.
• 01:24:32Right? Like, they threw a
• 01:24:33lot of money at it,
• 01:24:35but what's the product?
• 01:24:37So if we think about
• 01:24:38it as a product
• 01:24:39and interrogate it and say,
• 01:24:41well, how is AI being
• 01:24:42used here and do I
• 01:24:44like it? Do I do
• 01:24:45I trust
• 01:24:47x y z
• 01:24:48OpenAI,
• 01:24:49you know, as an organization?
• 01:24:51Can can we reframe
• 01:24:54its power over us
• 01:24:57by asking
• 01:24:59more specific questions as consumers
• 01:25:04about
• 01:25:05where did your model come
• 01:25:06from, what are your data
• 01:25:07sources, how do you do
• 01:25:08this, and how and I'll
• 01:25:10I'll stop with this thought
• 01:25:12and and maybe you can
• 01:25:14extrapolate.
• 01:25:16Where in the design process
• 01:25:17of
• 01:25:19large language model systems
• 01:25:21does patient centered care show
• 01:25:23up?
• 01:25:24How do they get designed?
• 01:25:26Or where
• 01:25:28where might we design them?
• 01:25:30Or say that, you know,
• 01:25:32maybe a hospital system says
• 01:25:33you have to design it
• 01:25:34with this in mind. You
• 01:25:36have to speak to consumers
• 01:25:37of care in your design.
• 01:25:39Can we push back as
• 01:25:41consumers
• 01:25:41and make greater demands and
• 01:25:43maybe slow this role?
• 01:25:47Yeah. I think so. I
• 01:25:49think that would be putting
• 01:25:50ourselves in the shoes of
• 01:25:51the patients,
• 01:25:52as I mentioned before.
• 01:25:56And
• 01:25:57that would
• 01:25:58require some accountability.
• 01:26:00And as much as, you
• 01:26:01know, people have
• 01:26:03tried to develop things that
• 01:26:05save clinicians time or
• 01:26:08reduce workload,
• 01:26:09I think we have to
• 01:26:11really evaluate and see whether
• 01:26:12they actually do because many
• 01:26:14times they don't. They add
• 01:26:15workload. They increase times.
• 01:26:17They don't do what they
• 01:26:19were intended to do. So
• 01:26:22we have to have some
• 01:26:23way of insisting.
• 01:26:30I think I wanna be
• 01:26:31respectful of everybody's time, and
• 01:26:33I wanna end sharply at
• 01:26:34seven, but I think we
• 01:26:35have time for one one
• 01:26:36final
• 01:26:38comment or question. Thank you
• 01:26:39so much. As I've been
• 01:26:39listening, you know, I was
• 01:26:40in clinic today all day
• 01:26:42feeling a little weighed down
• 01:26:44by the digital system and
• 01:26:45the paperwork demands and feeling
• 01:26:47a little
• 01:26:48despondent at the end of
• 01:26:49the day about my contribution.
• 01:26:51And, you know, I've been
• 01:26:52trying to think about, in
• 01:26:53general, the way I interact
• 01:26:55with the medical record and
• 01:26:56note writing, and
• 01:26:58I've I've been trying to
• 01:27:00wonder about how
• 01:27:02note writing as an empathic
• 01:27:04practice can be helpful to
• 01:27:05us and to our trainees.
• 01:27:06And I wonder how you
• 01:27:08see the role of AI
• 01:27:10in terms of opportunities for
• 01:27:12increasing or
• 01:27:13decreasing our sense of,
• 01:27:15connection with the patient on
• 01:27:17a human level in our
• 01:27:18documentation?
• 01:27:20Sorry. Connection with
• 01:27:22you know, the connection with
• 01:27:23the patient on a human
• 01:27:24level and sort of empathy
• 01:27:26building versus my feeling of
• 01:27:28being weighed down at the
• 01:27:30end of the day?
• 01:27:31Yes. I've I'm sorry you
• 01:27:33feel weighed down by all
• 01:27:34that.
• 01:27:36And AI is supposed to
• 01:27:37help that.
• 01:27:41But I do think, you
• 01:27:42know, I've we were just
• 01:27:43at the American Society of
• 01:27:44Bioethics and Humanities meeting, last
• 01:27:46week, and people have been
• 01:27:48studying this. So I think
• 01:27:49there'll be a lot to
• 01:27:51discover about what the effects
• 01:27:53are of AI in in
• 01:27:55these communications between patients
• 01:27:58and their clinicians. And some
• 01:28:00of the results were kind
• 01:28:01of surprising.
• 01:28:04So I'm not sure that
• 01:28:05I we know what those
• 01:28:06effects will be.
• 01:28:10And it it so far,
• 01:28:12it's you know, the jury's
• 01:28:13still out. But,
• 01:28:15I think that
• 01:28:17one thing that seems to
• 01:28:18be
• 01:28:19the case is that,
• 01:28:21at least right now,
• 01:28:23patients can tell
• 01:28:25the difference between AI generated
• 01:28:27message and their clinicians
• 01:28:29generated messages.
• 01:28:31So
• 01:28:32whatever the effects are, they'll
• 01:28:34know where it comes from
• 01:28:35one way or the other.
• 01:28:38And
• 01:28:39and so
• 01:28:40if they know that it's
• 01:28:42an AI generated message, then
• 01:28:44if they are seeking some
• 01:28:45kind of
• 01:28:47connection,
• 01:28:49then they
• 01:28:51will know that they're not
• 01:28:53getting it.
• 01:28:54On the other hand, I
• 01:28:55think some of the studies
• 01:28:57suggested that the,
• 01:28:59the patients didn't necessarily
• 01:29:01mind that there was an
• 01:29:02AI generated
• 01:29:04message if
• 01:29:05they thought a clinician
• 01:29:07or any clinician was reviewing
• 01:29:09the message
• 01:29:10for accuracy.
• 01:29:12So it may be that
• 01:29:13people are not really looking
• 01:29:14to those kinds of communications
• 01:29:16for,
• 01:29:18you know, a warm and
• 01:29:19fuzzy experience and that they're
• 01:29:21looking for that elsewhere,
• 01:29:23maybe in person.
• 01:29:26So I think we don't
• 01:29:27really know how those things
• 01:29:29will affect
• 01:29:31the relationship.
• 01:29:33But,
• 01:29:35I hope that it makes
• 01:29:37you, feel less weighed down
• 01:29:39in the future.
• 01:29:42Thank you so much. Thank
• 01:29:44thank you to to the
• 01:29:45everybody,
• 01:29:46who who participated and for
• 01:29:48your your questions and and
• 01:29:50comments. And thank you especially
• 01:29:51to doctor Choi. I know
• 01:29:52that Tom would have been
• 01:29:54really delighted with your talk
• 01:29:56and with the dialogue that
• 01:29:57you had. Thank you.
• 01:29:59I wanna
• 01:30:00extend
• 01:30:02the gesture of
• 01:30:04appreciation from the program for
• 01:30:05biomedical ethics. Some Yale swag
• 01:30:07Thank you. As is our
• 01:30:08tradition.
• 01:30:10Thank you very much.