“Wrong answers: When Simple Interpretations Create Complex Problems for Addiction Science Research and Policy”

Transcript

• 00:11Okay. Welcome everyone
• 00:14to General Medicine Noon Conference.
• 00:17CME code for today is
• 00:18five five nine zero five.
• 00:22Upcoming,
• 00:23retreat is, well, next one
• 00:26is May twenty ninth. I
• 00:27have a little bit of
• 00:27time on that. Educational retreat.
• 00:29Please watch out for information
• 00:31and opportunities
• 00:33to sign up.
• 00:35This is our weekly
• 00:37is it bouncing for you?
• 00:38It's not just my okay.
• 00:40Our weekly bouncing reminder for
• 00:42the, your FDACs.
• 00:45We are now entering the
• 00:46phase where the senior faculty
• 00:48are meeting and,
• 00:49trying to come up with
• 00:50good advice and suggestions for
• 00:52everybody and then watch out
• 00:53for opportunities to meet with
• 00:55your meet with your mentors.
• 01:00Upcoming research and progress in
• 01:02grand rounds.
• 01:03Next week,
• 01:05doctor Nikkayan will be speaking
• 01:07about interventional psychiatric treatments,
• 01:10and other modalities that we
• 01:12or at least they someone
• 01:13is using in clinic and,
• 01:14and in the hospital.
• 01:16And then next week, we'll
• 01:17have our section faculty and
• 01:19staff meeting.
• 01:23Disclosures.
• 01:26Okay.
• 01:28Excited now to, present doctor
• 01:30David,
• 01:31Fink,
• 01:32who, was originally,
• 01:34an undergrad,
• 01:35at University of California at
• 01:37San Diego.
• 01:38Oh, sorry. San Diego State
• 01:40University,
• 01:41in,
• 01:43biostatistics
• 01:44and epidemiology
• 01:45before getting his PhD at
• 01:47Columbia,
• 01:48where he focused on understanding
• 01:50what state level policies,
• 01:52are affecting,
• 01:53the care and out outcomes
• 01:55for patients who are using
• 01:56a variety of substances using
• 01:57a prescript looking at prescription
• 01:59drug monitoring programs at the
• 02:00state level as well
• 02:02as legalization of marijuana. So
• 02:03very timely,
• 02:05and important interventions to be
• 02:07studying.
• 02:08But when doing those types
• 02:10of studies, critical
• 02:12area of focus is the
• 02:14methods and understanding more about
• 02:16causal inference, which is something
• 02:17that a lot of us
• 02:18have, thought about and struggled
• 02:20with over the years.
• 02:23Specifically,
• 02:25how doctor Fink is applying
• 02:26those, is as as a
• 02:28substance use and psychiatric
• 02:30epidemiologist.
• 02:31His research broadly aims to
• 02:33develop and apply rigorous
• 02:35causal inference
• 02:36methodologies
• 02:37to study the causes of
• 02:38addiction and mental illness with
• 02:40a particular focus on estimating
• 02:42the effects of federal and
• 02:43state policies and programs.
• 02:45His research in mental health,
• 02:47substance use and health policy
• 02:49are united by a desire
• 02:50to not only understand,
• 02:52but reshape the structural, societal,
• 02:54and interpersonal factors that shape
• 02:56health and well-being over the
• 02:58life course.
• 02:59So we're really excited to
• 03:00have doctor Fink with us
• 03:01here at Yale in general,
• 03:04but also here at our
• 03:05noon conference to talk to
• 03:06us about,
• 03:09wrong answers when simple
• 03:11interpretations
• 03:12create complex problems for addiction
• 03:14science and policy.
• 03:15So welcome. Thank you.
• 03:22No.
• 03:23Thank you for the introduction.
• 03:26And thank you all for
• 03:27for being here on this
• 03:28pretty, rainy, crummy day, for
• 03:31and the people on Zoom
• 03:32who stayed home.
• 03:33So when I when I
• 03:35joined the department about a
• 03:35year and a half ago,
• 03:36I was I was pretty
• 03:37overwhelmed, I think, by
• 03:39the size of it. Even
• 03:40though I'd come from Columbia
• 03:41Psychiatry, which is about three
• 03:43hundred people, it just felt
• 03:44so different when you were
• 03:45there for a decade versus
• 03:46coming somewhere new.
• 03:48These Thursday meetings have been
• 03:49a great way for me
• 03:50to get to know people
• 03:51and know the department a
• 03:52little bit. So I really
• 03:53appreciate that. And this was,
• 03:55I was very happy to
• 03:56be asked to speak here
• 03:57and share my work with
• 03:59you.
• 04:00So the the title of
• 04:01my talk is wrong answers
• 04:02when simple interpretations
• 04:04create complex problems.
• 04:06But to set the stage
• 04:08for this talk, I think
• 04:09it's
• 04:09most important.
• 04:11And since people don't aren't
• 04:12familiar with me to to
• 04:13go back a step and
• 04:14actually actually talk a little
• 04:16bit how I got here,
• 04:17and that'll help explain, I
• 04:18think, where I'm at and
• 04:20where I I plan to
• 04:21kind of be heading. So
• 04:22if I start at the
• 04:24beginning, it really started with
• 04:25the realization that we learn
• 04:27about in class all the
• 04:28time in school when you're
• 04:29you're studying, which is how
• 04:30systems and politics affect people's
• 04:32health.
• 04:33But the part that really
• 04:34stuck with me is I
• 04:35hadn't really seen that before.
• 04:37And on top of not
• 04:38seeing it before, I didn't
• 04:39understand how divorced evidence was
• 04:41from policies
• 04:42and how much worse that
• 04:43was when you're talking about
• 04:44stigmatized populations, which is something
• 04:46many of us, I think,
• 04:47deal with. So I got
• 04:48to become familiar with that
• 04:49while working on San Diego's,
• 04:51safe syringe program.
• 04:54And so
• 04:55I was really ignorant of
• 04:56how much politics would affect
• 04:57day to day life, for
• 04:59people who used injection drugs
• 05:01as well as, even the
• 05:02people who tried to help
• 05:04people who use injection drugs.
• 05:07And so in San Diego,
• 05:09we had to operate under
• 05:10a public health state of
• 05:11emergency, which mean every week
• 05:12someone had to vote to
• 05:13say this emergency was still
• 05:15happening or we couldn't even
• 05:16operate.
• 05:17Even on when we did
• 05:18operate, we operated two hours
• 05:20a day,
• 05:21for two days of a
• 05:22week.
• 05:23So four hours total,
• 05:24on a one for one
• 05:26exchange for fifty needles, which
• 05:27meant that basically if anything
• 05:28happened in somebody's life and
• 05:30they couldn't make it, they
• 05:31were not getting needles that
• 05:32week because you couldn't exchange
• 05:33for other people really. And
• 05:34one for one meant if
• 05:35you didn't have any, the
• 05:36person who needed it most
• 05:38couldn't even get any.
• 05:39And even with those kind
• 05:40of restrictions,
• 05:42there were constant news reports
• 05:43and protesters and everybody else
• 05:45saying how unhappy they were
• 05:46about us even being present,
• 05:47which mean we couldn't provide
• 05:49confidentiality and and the services
• 05:51that we really were trying
• 05:52to provide.
• 05:53And even on the best
• 05:54days,
• 05:55and everything was working perfectly,
• 05:57our one of our jobs
• 05:58was to get people into
• 05:59detox. And back in the
• 06:01early two thousands when I
• 06:02worked there, buprenorphine was almost
• 06:04unheard of. It was very
• 06:06rare to see.
• 06:07Methadone was at two places
• 06:09in a town of thirty
• 06:10three million or so. So
• 06:11there was only cold turkey
• 06:13detox. That was all that
• 06:14we had access to. And
• 06:15there was twenty beds in
• 06:16the whole city. So you
• 06:17kinda had to let the
• 06:18stars align to make things
• 06:19work. So we had this
• 06:21population you were trying to
• 06:22help,
• 06:23and it seemed like every
• 06:24single system was working against
• 06:25you.
• 06:26And so that's kind of
• 06:28where I wanted to head
• 06:29today a little bit, to
• 06:30understand why that happens.
• 06:32But one of the the
• 06:33big takeaways that I learned
• 06:34from this was
• 06:35the importance of
• 06:37learning from the population that's
• 06:39affected and having conversations and
• 06:40interactions with them and how
• 06:41I think some of the
• 06:42best questions come from that,
• 06:44from those interactions. And so
• 06:45one of the things that
• 06:46I became very aware of
• 06:47when I started working there
• 06:48was abscesses. I had not
• 06:50been exposed to abscesses, not
• 06:51working in that population before.
• 06:53I didn't realize how prevalent
• 06:54they were. I didn't understand
• 06:55how people dealt with them.
• 06:56You're dealing with a population
• 06:57that did not trust any
• 06:58authority and that included the
• 06:59medical community for the most
• 07:00part. So there's a lot
• 07:02of self treatment happening. I
• 07:03one conversation sticks with me
• 07:05even particularly where an individual
• 07:07tried to take care of
• 07:08an abscess by pulling out
• 07:10the fluid with a syringe,
• 07:11and it looked like heroin,
• 07:12so they decided to mainline
• 07:14inject it again.
• 07:16So there was a lot
• 07:17of misinformation that was happening
• 07:18and a lot of choices
• 07:20that, were happening because there
• 07:22was systems that weren't in
• 07:23place. And so I'm particularly
• 07:24proud. It was my first
• 07:25paper that I wrote. It
• 07:27really brought light to self
• 07:29care or of wounds, which
• 07:30was something that wasn't in
• 07:31the literature at that point.
• 07:34You know, we found that
• 07:35about half of people in
• 07:36San Diego or half of
• 07:37clients of syringe exchange programs
• 07:38were self treating. Most of
• 07:40that was with, self lancing
• 07:41with about a third of
• 07:43them doing that, and then
• 07:44about ten percent using,
• 07:46illegally purchased antibiotics. So it
• 07:48was something that was prevalent
• 07:49and it's become a lot
• 07:50more discussed, but at the
• 07:51time it really didn't seem
• 07:53to be part of it.
• 07:53And I think that
• 07:55one of those things that
• 07:56we get away from the
• 07:56wrong answers is by having
• 07:58conversations with people most affected.
• 08:01So this is a place
• 08:02where I really entered my
• 08:03research, learning about the struggles
• 08:05and experiences of people with
• 08:07addictions,
• 08:08the risk factors that were
• 08:09in place, many of them
• 08:10completely preventable, and
• 08:12the the national and local
• 08:14systems and policies that really
• 08:16worked against,
• 08:18kinda changing the health environment
• 08:19where people worked.
• 08:21So the question then
• 08:23becomes, why does why are
• 08:24the systems in place? And
• 08:26this really includes both research
• 08:27and political systems,
• 08:29and how do they produce
• 08:30these bad policies or harmful
• 08:32policies even?
• 08:35And so that's what I'm
• 08:36gonna talk about here today.
• 08:37I'm gonna bring up two
• 08:38different, points and two different
• 08:40intertwined issues
• 08:41and give examples from my
• 08:43own research here. So one
• 08:44is
• 08:45the decision that often happens
• 08:46in prioritizing
• 08:47policy and research that focuses
• 08:49on identifying and intervening with
• 08:51high risk populations.
• 08:53It's a general approach. It's
• 08:54what we tend to do
• 08:55where especially if there's a
• 08:57very high risk group, there's
• 08:58a a tendency to other
• 08:59that group and say they're
• 09:00not part of our population,
• 09:02and we can intervene with
• 09:03just them instead of understanding
• 09:05that that part of the
• 09:06population is part of the
• 09:07whole risk distribution.
• 09:08We need to see the
• 09:09whole risk distribution. And so,
• 09:11but but that's something that's
• 09:12rarely done, and especially in
• 09:13addiction,
• 09:14we don't tend to think
• 09:15this way.
• 09:17The second issue is more
• 09:18of a scientific issue. It's
• 09:20an approach to causation that's
• 09:21really prioritizing what's easy to
• 09:23measure, what's, easily accessible,
• 09:26the quantitative data, ignoring some
• 09:28of those harder to measure
• 09:29and quantify or
• 09:31quantify, qualitative factors that matter
• 09:33so much. And I'll introduce
• 09:34what's called the McNamara fallacy,
• 09:36and how that plays into
• 09:38this work in the second
• 09:39half of this talk.
• 09:41So I'm gonna try to
• 09:42cover all this in the
• 09:42next twenty five minutes or
• 09:45so.
• 09:46And as I said, give
• 09:47some examples from my work
• 09:48and how we've kind of
• 09:48looked into this. So if
• 09:49we start with the high
• 09:50risk prevention strategy,
• 09:53so for those who are
• 09:54not familiar with this, gentleman
• 09:56here, this is Jeffrey Rose.
• 09:57He's been been one of
• 09:58the most influential people, I
• 09:59think, in my my work.
• 10:00He wrote this brilliant book
• 10:01on the strategies of preventative
• 10:03medicine.
• 10:05And,
• 10:07it's one of the big
• 10:08takeaways of the book was
• 10:09that the difference in prevalence
• 10:11of an outcome in different
• 10:12populations is really due to
• 10:14the different states
• 10:15of the health of the
• 10:16parent communities.
• 10:18The slide is a lot,
• 10:19so I'll try to kinda
• 10:20walk you through it because
• 10:21I think it's important to
• 10:22explain what we did to
• 10:23demonstrate a lot of this
• 10:24work.
• 10:26And so what you what
• 10:26you have here is in
• 10:27the the light orange or
• 10:29tan is, a population's distribution.
• 10:31And in that far right
• 10:33corner, we have that part
• 10:34that's at the highest risk.
• 10:36Right? And so a lot
• 10:37of times when we think
• 10:38about what to do with
• 10:39this situation and what policy
• 10:40makers tend to do, we
• 10:42tend to focus on that
• 10:43far right dark orange corner
• 10:45and say, how can we
• 10:45move them out? How can
• 10:47we move that group over?
• 10:48So a lot of that's
• 10:49done through identification of high
• 10:50risk people and then treatment
• 10:51of high risk people or
• 10:53identification and linking to another
• 10:55care.
• 10:56And it's less often done,
• 10:58especially with stigmatized,
• 11:00conditions and outcomes. Thinking about
• 11:02how we can shift that
• 11:03whole distribution, which is more
• 11:04of that population based approach.
• 11:06So how can we take
• 11:08the people that are normal
• 11:09and bring them to low?
• 11:10How can we take people
• 11:10that are high and bring
• 11:11them to normal? And at
• 11:12the same time when we
• 11:13do that, we're also moving
• 11:14the high group out.
• 11:16And this is, an approach
• 11:18that is,
• 11:20as I said, it's it's
• 11:21it is used. We see
• 11:22it frequently used, with
• 11:25more common conditions and less
• 11:26stigmatized conditions, things like blood
• 11:28pressure. You know, you would
• 11:29definitely have this be a
• 11:30population based approach usually.
• 11:32But things like addiction, it
• 11:33usually is not.
• 11:35And in in the example
• 11:36that I'm gonna give, one
• 11:38of the ways that we
• 11:38went to look at this
• 11:39was actually something that comes
• 11:41up frequently with firearms and
• 11:43mental illness.
• 11:44And so a predictable cycle
• 11:46that happens every time there
• 11:48seems to be a mass
• 11:49shooting is a discussion about
• 11:50the role of mental illness
• 11:51and firearm deaths and the
• 11:52need to focus on that
• 11:53high risk group compared to
• 11:55your population approach, which is
• 11:56gonna move the whole distribution.
• 11:59So in a collaboration with
• 12:00some colleagues at at Columbia
• 12:02and NYU, which include, Magdalena
• 12:04Serta, who's will be joining
• 12:06the school of public health
• 12:07as the new chair of
• 12:08chronic disease epi.
• 12:10So we carried out a
• 12:11study to better understand the
• 12:12narrative of firearm and mental
• 12:14illness
• 12:14and to compare a targeted,
• 12:16high risk approach versus more
• 12:18of the population based approach,
• 12:20in an agent based model.
• 12:22And so
• 12:23for those who are not
• 12:24familiar with an agent based
• 12:25model,
• 12:27basically, what this is is
• 12:28it's a way to model
• 12:29dynamics of a changing system
• 12:30to better understand how if
• 12:32you shift or move one
• 12:33piece of it, how you
• 12:34have downstream effects.
• 12:36And you do that by
• 12:37simulating agents, which are basically
• 12:40individuals. And those individuals,
• 12:42they live within a environment.
• 12:43They have characteristics of that
• 12:45environment. They interact
• 12:51with each other. They interact
• 12:51with their the sit the
• 12:51situation. And you can do
• 12:51it for a city like
• 12:51we did for the adult
• 12:52population of New York.
• 12:54And so you place these
• 12:55individuals
• 12:56within their communities. And as
• 12:57people become eighteen years old,
• 12:59they, you know, move through
• 13:01different risk strata and different
• 13:02occurrences.
• 13:03And they live their life
• 13:05cycle in that way. And
• 13:06so this is all done
• 13:07through a bunch of equations
• 13:08essentially.
• 13:09And this is what those
• 13:11schematic looks like. It's very
• 13:12complex. It's not the simple
• 13:14one one cause, you know,
• 13:15one outcome thing. And this
• 13:17is a lot to look
• 13:18at. So if we kinda
• 13:20zoom in here on the
• 13:21social network characteristics, you can
• 13:22see that agents form and
• 13:24and dissolve social ties. They
• 13:26have friends who were who
• 13:27are perpetrators and victims of
• 13:29violence and some who own
• 13:30firearms. And with each iteration,
• 13:32the agents move through their
• 13:34lives essentially.
• 13:36And so to look at
• 13:37these different prevention strategies and
• 13:39what and demonstrate their utility
• 13:41in this kind of situation,
• 13:44We looked at three different
• 13:45groups for disqualification.
• 13:47So we looked
• 13:48at the the first group
• 13:49is the low prevalence group,
• 13:50and this actually does include
• 13:51the psychiatric hospitalizations.
• 13:54Sorry, I shouldn't say convictions.
• 13:55It should be just psychiatric
• 13:56hospitalizations,
• 13:58as well as people that
• 13:59are alcohol related misdemeanors. And
• 14:00so this group is the
• 14:02lowest prevalence group in the
• 14:03population. It's about a quarter
• 14:04of a percent.
• 14:07The next group is the
• 14:08moderate prevalence group, and so
• 14:09this is drug misdemeanor convictions
• 14:11and domestic violence restraining
• 14:13orders.
• 14:14And these people make up
• 14:15about one percent of a
• 14:16population. And then on the
• 14:19last one, we have the
• 14:20high prevalence group, which is
• 14:22comprised of all felony convictions
• 14:24and misdemeanor,
• 14:26convictions. And so this is
• 14:27the largest group. It's about
• 14:28two point five percent. And
• 14:29so we're comparing disqualifications in
• 14:31these different groups, which is
• 14:33what is often the discussion
• 14:34around firearm violence and how
• 14:36to prevent it versus a
• 14:37very population based approach, which
• 14:38is increasing prices.
• 14:40So just basically increasing prices
• 14:41on the firearms and ammunition
• 14:43and what can we do
• 14:44with that? Before you go.
• 14:46Yes, please. I can answer.
• 14:48Disqualifications?
• 14:49Disqualifications. They cannot purchase guns.
• 14:52So
• 14:53Exclude they're not allowed to
• 14:54buy a gun. Correct.
• 14:56Correct. However, we also do
• 14:58model whether they can get
• 14:59a gun illegally. So if
• 15:00you have a friend
• 15:01who has a firearm
• 15:03in a year and you
• 15:04are a perpetrator, you can
• 15:05get the firearm through that
• 15:06connection.
• 15:07So all those connections are
• 15:09modeled still too.
• 15:11And so the way you
• 15:12actually prove that you understand
• 15:13the model is by validating
• 15:15it by what happened in
• 15:16reality, and the goal is
• 15:17to have them match up
• 15:18and they do. So we
• 15:19did look at illegally purchased
• 15:20firearms and how that is
• 15:21affected.
• 15:22I don't show that here,
• 15:23but how that is affected
• 15:25by prices going up and
• 15:26things like that. So we
• 15:27really do try to model
• 15:28the whole system.
• 15:30It's process.
• 15:32So, so the first thing
• 15:33to kinda take from this
• 15:34is that the baseline firearm
• 15:35high on-site rate in New
• 15:36York was four per hundred
• 15:38thousand persons. So the goal
• 15:39of the first step was
• 15:40just to decrease it by
• 15:41five percent. What could we
• 15:42do?
• 15:43So first is removing the
• 15:45low prevalence group, which included
• 15:46that psychiatric hospitalizations. You remove
• 15:48every single firearm from that
• 15:49low prevalence group, you can
• 15:50only reduce it by two
• 15:51percent.
• 15:53That's the most you could
• 15:54do if you wanna do
• 15:55effects firearm homicides.
• 15:58In the moderate group, if
• 15:59you removed it from twenty
• 16:00five percent, you could get
• 16:01to that five percent level.
• 16:04In the high prevalence group,
• 16:05it's even less twelve percent.
• 16:07And then if we increase
• 16:08price, you'd increase price by
• 16:10just eighteen percent and you'd
• 16:11hit that same five percent.
• 16:12So having a lot of
• 16:13people at that lower risk
• 16:15is going to have a
• 16:16bigger impact when you're affecting
• 16:18all of them versus the
• 16:19few people at high risk.
• 16:20If you combine all three
• 16:22of them, you would get
• 16:22to a twelve percent reduction
• 16:24at this.
• 16:26And then just to show
• 16:28how
• 16:30how the the next stage
• 16:31of it kinda goes, you
• 16:32can look at different levels
• 16:33of this. And this is
• 16:34the the
• 16:35interesting part about these models
• 16:36is once you build them,
• 16:37you can look at them
• 16:38a lot of different ways.
• 16:39So if we wanna see
• 16:39a bigger example, what we
• 16:41basically start to see is
• 16:42that price becomes the only
• 16:43thing that's influential.
• 16:45You know, even if we
• 16:46remove a hundred percent of
• 16:47the firearms again from that
• 16:49moderate group,
• 16:50we're reaching sixteen percent. So
• 16:51we can't even hit that
• 16:52twenty five. And a hundred
• 16:53percent disqualification would never happen
• 16:54by the way, just to
• 16:55be clear. Like, this is
• 16:56a hypothetical world. This isn't
• 16:58reality. We would never be
• 16:59able to take all of
• 17:00them away. So even if
• 17:01you did take every gun
• 17:02away from this group or
• 17:03firearm, it still would not
• 17:05have the desired effect of
• 17:06anything above sixteen percent.
• 17:09The high prevalence group, you
• 17:11could get a little bit
• 17:11closer, but you are gonna
• 17:12top out pretty soon after
• 17:14that. And you again, you
• 17:15see that price is really
• 17:16the only thing.
• 17:18And unfortunately, despite this kind
• 17:19of evidence, the contrary that
• 17:21mental illness is not driving,
• 17:23these instances,
• 17:24we hear the same debate
• 17:25that's really continuing after every
• 17:27single mass shooting. How about
• 17:28the folk need to focus
• 17:29on mental illness?
• 17:31And we definitely need better
• 17:32access to mental illness. I
• 17:33would or treatment. I would
• 17:34never say otherwise. But the
• 17:36research shows that focusing interventions
• 17:38on this group or any
• 17:39group is going to be
• 17:40insufficient,
• 17:42in this kind of situation
• 17:44where a lot of the
• 17:44cases are coming from those
• 17:46at lower risk. And so
• 17:47it's a need to think
• 17:48about the whole population
• 17:50and their distribution.
• 17:53So this example really demonstrates,
• 17:55again, the difference in how
• 17:56we think about the high
• 17:57risk prevention strategy versus the
• 17:59population based strategy when we're
• 18:00when we're thinking about policies.
• 18:02And there's a lot of
• 18:03examples of this. The high
• 18:05risk prevention strategy being promoted
• 18:07over that population based approach.
• 18:10And
• 18:11this is particularly true when
• 18:12we looked at stigmatized outcomes.
• 18:14And then I'm gonna so
• 18:15I'm gonna pivot here to
• 18:16focus on the next stigmatized
• 18:17outcome, which is more of
• 18:18the the other topic of
• 18:19this stuff, which is looking
• 18:20at, drug use, addiction, overdose.
• 18:23And so if we look
• 18:24specifically at overdose,
• 18:27high risk,
• 18:28prevention strategies are often prioritized.
• 18:30But
• 18:32as we talk about that,
• 18:33I wanted to introduce the
• 18:34next topic I said I
• 18:35was going to, which is,
• 18:38the the two common
• 18:40in both policy making and
• 18:42in research in particular.
• 18:44We have a tendency to
• 18:45focus on those easily measured
• 18:47and to easy to measure
• 18:49metrics instead of those meaningful,
• 18:50harder to measure items.
• 18:53And so this is the
• 18:54basis of the McNamara fallacy.
• 18:56So the gentleman who gets
• 18:58the honor of having this
• 18:59named after him is Robert
• 19:00McNamara. He was the US
• 19:01secretary of defense during the
• 19:02Vietnam War.
• 19:04And there's a lot of
• 19:04different reasons this has been
• 19:06attributed to him. I think
• 19:07the one that I see
• 19:08most often
• 19:09is that during the war,
• 19:10he became highly focused on
• 19:12the metrics of deaths, and
• 19:14he thought that you could
• 19:14win a war of attrition.
• 19:16So if you simply counted
• 19:17how many people you killed
• 19:18versus how many people on
• 19:20your side died, eventually, you
• 19:22would have the winner.
• 19:24And that never happens. And
• 19:25the reason that never happens
• 19:27is because focusing only on
• 19:29killing another population
• 19:30is going to destroy
• 19:32any
• 19:33goodwill or any other feelings
• 19:35that could exist. Any of
• 19:35the rural population that is
• 19:37affected by this and the
• 19:38people you're supposedly trying to
• 19:39help,
• 19:40is not
• 19:41they're not coming to your
• 19:42side, essentially. So you're missing
• 19:44that harder to measure qualitative
• 19:45factor, which is the attitudes
• 19:47on the ground and how
• 19:47people felt.
• 19:49And so this is often
• 19:50stated in, three parts. So
• 19:52the fallacy basically says that
• 19:54you measure what's easy to
• 19:55measure, you disregard that which
• 19:57can't easily be measured,
• 19:58and then you assume that
• 20:00whatever can't be measured is
• 20:01unimportant and you can even
• 20:02go step further. It doesn't
• 20:03even exist.
• 20:05And it basically says that
• 20:06focusing on metrics leads to
• 20:08a very narrow view and
• 20:09ignores that complexity and the
• 20:11crucial
• 20:11intangible factors
• 20:13that are gonna result in
• 20:14poor long term strategies.
• 20:16And I'd argue that almost
• 20:17all of the policy mishaps
• 20:19that have happened during the
• 20:20overdose crisis
• 20:23are fell victim to this
• 20:24fallacy,
• 20:25and we'll kind of walk
• 20:26through that here. One of
• 20:27the easiest ways to see
• 20:28that is really the focus
• 20:30of the
• 20:31of the crisis. So the
• 20:32metrics just were so easy
• 20:34in this case. You had,
• 20:35you know, opioid prescriptions. If
• 20:37you looked at them, dispensed
• 20:38between ninety nine and twenty
• 20:39thirteen. And we've all kind
• 20:40of seen these figures before.
• 20:41This isn't anything new.
• 20:42And when you overlay it,
• 20:43you just get such a
• 20:44perfect picture. And so I
• 20:46think this became the focus
• 20:48of the easiest to measure
• 20:49metric,
• 20:49which was supply. How do
• 20:51I affect supply?
• 20:53And you really see that
• 20:54in the policies that came
• 20:55into effect. The first policies
• 20:57sorry. This slide isn't the
• 20:58easiest to see, but this
• 21:00report came out in twenty
• 21:01eleven from the White House,
• 21:04Office National Drug Control Policy,
• 21:06and they put forward four
• 21:07different policies.
• 21:09First is education,
• 21:12educating patient providers on the
• 21:13risk of opioids.
• 21:15The second was advancing prescription
• 21:16drug monitoring programs.
• 21:18The third was, increasing access
• 21:20to proper disposal of unused
• 21:21medications. And third and finally,
• 21:25increasing enforcement for illegal
• 21:27prescriptions.
• 21:29But at the time, there
• 21:30really was no evidence to
• 21:31support these claims. And not
• 21:33only that, the evidence that
• 21:34did exist was looking at
• 21:36prescription opioid supply.
• 21:38And so
• 21:39one of the the
• 21:41papers that we did was
• 21:42question this and ask,
• 21:44do we care about prescription
• 21:45opioid supply or do we
• 21:46care about deaths? Do we
• 21:47care about actual some measure
• 21:49of outcomes?
• 21:50And this isn't necessarily a
• 21:51really hard to measure metric,
• 21:53but I think it still
• 21:54illustrates,
• 21:55that this was one of
• 21:56the key metrics that were
• 21:57just ignored,
• 21:58in a lot of the
• 21:59early policies.
• 22:00And so at the time
• 22:01this article came out, there
• 22:02were seventeen papers looking at
• 22:04prescription drug monitoring programs and
• 22:06death. There was really low
• 22:07grade evidence, which means there
• 22:08was conflicting results. It means
• 22:11that there was risk to
• 22:12bias in a lot of
• 22:13them,
• 22:13and they still
• 22:15had just moderate,
• 22:16evidence that it reduced prescription
• 22:18opioids
• 22:19deaths. But the real concern
• 22:20that came out of it
• 22:21was that it was shown
• 22:22to increase heroin related deaths
• 22:24in a much more
• 22:25rigorous fashion.
• 22:27And this was the first
• 22:28one of the first papers
• 22:29that really brought light, I
• 22:30think, to the unintended consequences
• 22:31of these policies at a
• 22:32at a large level.
• 22:35And then it became a
• 22:36regular occurrence. And I think
• 22:37this paper was a brilliant
• 22:39paper that was done by,
• 22:41Pitt and colleagues.
• 22:42It's a systems dynamic model,
• 22:44which is kinda like an
• 22:45agent based model. There's a
• 22:46lot of thinking about different
• 22:47creating a whole society and
• 22:49then trying these different interventions.
• 22:52And if we looked at
• 22:53this article
• 22:55and four of these outcomes
• 22:56here, these were really focused
• 22:58on prescribing.
• 22:59Again,
• 23:00we saw that four of
• 23:01the outcomes, which was chronic
• 23:02pain reducing chronic pain prescribing,
• 23:05which we thought as a
• 23:06tapers, drug rescheduling, prescription drug
• 23:08monitoring program, and then drug
• 23:10reformat reformulation
• 23:11such as,
• 23:13abuse deterrent, OxyContin,
• 23:15that all these actually
• 23:17reduced,
• 23:19prescription opioid deaths, but they
• 23:20were completely offset by heroin
• 23:21deaths. It was a complete
• 23:22lack of understanding about the
• 23:24complexity of what was occurring.
• 23:25And there's
• 23:26now, you know, set almost
• 23:28ten years of papers maybe
• 23:29or maybe not that much
• 23:30that have all, you know,
• 23:32demonstrated the same thing. The
• 23:33reformulation was extremely harmful. And
• 23:35it's a lack of understanding
• 23:36the whole system, the complex
• 23:38network of the system, how
• 23:39they all work together and
• 23:40understanding the unintended effects of
• 23:41these.
• 23:43And so in an attempt
• 23:44to kind of start to
• 23:45challenge this and push back
• 23:47against it,
• 23:48one of the some of
• 23:49the work that we did
• 23:50was try to quantify some
• 23:51of the harder measure stuff.
• 23:53Yes.
• 23:54Yes.
• 23:55I think this is super
• 23:56important and not not surprising,
• 23:58but But I feel like
• 23:58sometimes the time horizon
• 24:00is wrong because you can
• 24:01imagine a world where the
• 24:02short term
• 24:04people moving from prescription abuse
• 24:06to sort of Yes. Ethanol
• 24:08sort of sees sort of
• 24:09offsets the the ending benefit
• 24:10of the debt. And maybe
• 24:12the long term
• 24:13sort of fewer entrance into,
• 24:14like, the risk pool if
• 24:16there are fewer prescription opioids
• 24:17as sort of as, like,
• 24:18a gateway. Hundred percent.
• 24:20Have any of these studies
• 24:21tried to, like I mean,
• 24:21clearly, that isn't maybe enough
• 24:23time to do that. Is
• 24:24there a way or has
• 24:25there any modeling of, like,
• 24:27potentially a long term benefit
• 24:28even if there's no Yes.
• 24:30So, yes, there has been.
• 24:31So this was the first
• 24:32paper that was done by
• 24:33this group. They published a
• 24:35paper, I think, two years
• 24:36later, in twenty twenty that
• 24:38did different time horizons.
• 24:40This was a five year
• 24:40time horizon. They also did
• 24:42a ten year time horizon.
• 24:43And when you hit the
• 24:44ten years, you start to
• 24:45see exactly what you're saying,
• 24:46where prescription drug monitoring programs
• 24:48even started to reduce deaths
• 24:49at that point. You just
• 24:51had to survive that ten
• 24:52year,
• 24:53harmful effects period to get
• 24:55to the benefit one. And
• 24:56and actually, I think a
• 24:58lot
• 24:59of the discussion around that
• 25:00what around all of this
• 25:02is that any of these
• 25:03programs, I think, could have
• 25:04been helpful, and that's something
• 25:05we can definitely talk about
• 25:06is that I think any
• 25:07of them could have been
• 25:08helpful if they were done
• 25:09differently, and it wasn't just
• 25:10so hyper focused on one
• 25:12aspect of it instead of
• 25:13thinking about the individual and
• 25:15the and understanding people and
• 25:16having them involved in it.
• 25:18Right?
• 25:19And so one of one
• 25:20of the things that we
• 25:21tried to quantify and look
• 25:22at was the other side
• 25:23of,
• 25:24what we started hearing a
• 25:25lot about, which was socioeconomic
• 25:27situations and how that could
• 25:28play into it. You know?
• 25:31And so when we looked
• 25:32at this systematic review,
• 25:34I enjoy systematic reviews. If
• 25:36you can't tell, I I,
• 25:37I like to do one
• 25:37every couple years. So it's,
• 25:39you know, these are topics
• 25:40you're interested in, I'm I'm
• 25:41available.
• 25:43But, one of the things
• 25:43we looked at was socioeconomic
• 25:45determinants of overdose deaths and
• 25:46really to understand what was
• 25:47the literature there, because that's
• 25:48a much harder to measure
• 25:49thing. It's much harder to
• 25:50measure.
• 25:52One of the so in
• 25:53this study, we found
• 25:54twenty seven studies had done
• 25:56this.
• 25:57And so that's a much
• 25:58more than they had looked
• 25:59at PDMPs in death. So
• 26:00this is a much more,
• 26:02study topic.
• 26:03And the results were universal
• 26:05across the board. Socioeconomic
• 26:07situations affected county level overdose
• 26:10rates,
• 26:11on every study. It didn't
• 26:12matter what what measure you
• 26:14looked at. It wasn't it
• 26:14didn't matter if you're looking
• 26:16at income inequality or if
• 26:17you were looking at poverty
• 26:18level. They all had the
• 26:19same effect.
• 26:21It was one of the
• 26:21most consistent findings I think
• 26:23I've ever found.
• 26:26And yet, I I have
• 26:27never heard a policy maker
• 26:28say, let's increase universal basic
• 26:30income to affect the overdose
• 26:31crisis or job training programs
• 26:33or anything to that extent.
• 26:34It's something we just don't
• 26:35hear as much.
• 26:37But one of the interesting
• 26:38things that we did is
• 26:39take this a step further.
• 26:42And so we then combine
• 26:44the two. So what's the
• 26:45role of socioeconomic situations and
• 26:46prescription opioids?
• 26:48And this was a really
• 26:48fascinating paper for me, because
• 26:50it didn't produce the effects
• 26:52I thought it would, but
• 26:53then it made sense.
• 26:55So basically, what we found
• 26:56in this paper
• 26:58was that
• 26:59in highly deprivized,
• 27:02environments, counties that had the
• 27:04most economic inequality, that had
• 27:07the highest federal poverty rates,
• 27:10prescription opioid supply had no
• 27:12role. It was completely unassociated.
• 27:14It was a pretty strong
• 27:15finding. It was it was
• 27:16pretty consistent across
• 27:18those locations as well.
• 27:20In in in places where
• 27:21there was much
• 27:23lower,
• 27:24poverty and much less income
• 27:26inequality, all of a sudden
• 27:27prescription opioid supply was very
• 27:28important.
• 27:29So, again, it wasn't what
• 27:31I think we I expected
• 27:33to find, but it actually
• 27:33started a story could start
• 27:35to come together,
• 27:37in understanding how these two
• 27:39work together and that highly
• 27:40deprivised area places like prescription
• 27:42opioid supply was just one
• 27:43more thing, you know, that
• 27:44was already affecting them.
• 27:46Whereas in in places that
• 27:48were
• 27:49doing better overall, this became
• 27:51something much more impactful. And
• 27:53so now if you're looking
• 27:53at all the policies that
• 27:54were focused on reducing supply,
• 27:56even if they all worked,
• 27:57you might have only been
• 27:58affecting one segment of the
• 27:59population because you didn't understand
• 28:01the problem.
• 28:02And so I think that's
• 28:03a a really important piece
• 28:05of this, and we really
• 28:06need more literature to understand
• 28:07this and actually capture these
• 28:08harder to measure
• 28:10metrics in a better way
• 28:11because I don't even think
• 28:11this is perfect. And I'm
• 28:13gonna continue to pull out
• 28:15my own research and my
• 28:16own shortcomings in this, and
• 28:17that's gonna kind of be
• 28:18the the next part of
• 28:19this talk is
• 28:21to understand
• 28:22how we continue to do
• 28:23this in pharmacoepi studies,
• 28:25and where the McNamara fallacy,
• 28:27I think, continues to live.
• 28:29And so in this past
• 28:30year, I received the r
• 28:31zero zero to look at
• 28:32buprenorphine treatment outcomes,
• 28:34in real world data. And
• 28:36so we are looking at
• 28:37VA EHR data. We're looking
• 28:39at what happened during COVID
• 28:41nineteen.
• 28:42And,
• 28:43you know, did telehealth increasing
• 28:44telehealth use affect people that
• 28:46were initiating buprenorphine? Did it
• 28:47affect the long term outcomes
• 28:49of people that had been
• 28:49on it a long time?
• 28:51And when we started putting
• 28:52this grant together, I looked
• 28:53for all the outcomes that
• 28:54I could find, in these
• 28:56kind of studies.
• 28:57And I found four that
• 28:59were most commonly used. So
• 29:01with almost
• 29:02no exceptions,
• 29:04and my study is not
• 29:05an exception, the primary outcome
• 29:06is always a hundred and
• 29:07eighty day retention in care.
• 29:09This is the metric that
• 29:10is used most often. The
• 29:11problem is is this is
• 29:12not a health metric, this
• 29:14is a process metric.
• 29:16We don't inherently care that
• 29:18someone's in care. If we
• 29:20care that they are stable,
• 29:21maybe we can make that
• 29:22statement, but that's not what
• 29:23that is necessarily measuring. And
• 29:25so that's the first one
• 29:26we see. And then a
• 29:27couple
• 29:28metrics that I was able
• 29:29also find is a list
• 29:30of drug use and toxicology
• 29:32data,
• 29:33opioid use specific hospitalizations,
• 29:35fatal overdoses.
• 29:38And there's so there's many
• 29:39reasons, I think, that a
• 29:41patient might seek out care,
• 29:43and we know these reasons.
• 29:44And some of them are
• 29:45on here. There's definitely a
• 29:47a desire to stay alive.
• 29:49That is one reason that
• 29:50drive that brings people into
• 29:52treatment.
• 29:52There's a desire to stop
• 29:54using
• 29:55drugs.
• 29:57And so there is a
• 29:58piece of that that can
• 29:59be found maybe in the
• 30:00toxicology data.
• 30:01But it's incomplete. And I
• 30:02think that that can be
• 30:04seen a little bit more
• 30:05when we actually look ask
• 30:06questions about what patients want.
• 30:10And so I found a
• 30:10couple of systematic reviews that
• 30:12were focused more on, patient
• 30:14goals.
• 30:15And when we look at
• 30:16these, we see again, there
• 30:17is some overlap of the
• 30:19the challenge that,
• 30:21that I became aware of
• 30:22is that
• 30:23even the things we are
• 30:24measuring were not clear
• 30:26in how we should measure
• 30:27them. And I think the
• 30:28first example of that is
• 30:29really good, the treatment related
• 30:30goals.
• 30:32When you looked at studies,
• 30:33there's one study in particular
• 30:34that asked patients about their
• 30:35goals.
• 30:36And about seventy percent said
• 30:37remain in treatment,
• 30:39while two thirds said to
• 30:40get off of buprenorphine.
• 30:42So the majority of them
• 30:43actually their one of their
• 30:44main goals was getting off
• 30:45of them. And now, obviously,
• 30:46these are thinking maybe more
• 30:47of a longer term period
• 30:48like a year or I
• 30:50don't know the exact timeline
• 30:51of what that would be.
• 30:54But
• 30:55they're using retention and goal
• 30:57in care might in and
• 30:58of itself not even be
• 31:00a patient centered goal,
• 31:02in that sense. There's, again,
• 31:04substance use related goals of
• 31:05avoiding withdrawal. Those could be
• 31:06seen in some of this.
• 31:08But the bottom one is
• 31:09completely absent,
• 31:10I think from almost any
• 31:11study that's using,
• 31:12administrative claims data.
• 31:15How to measure living a
• 31:16normal life, stability, reduce criminal
• 31:18activity, improved housing, employment, improved
• 31:20social and familial relationships. These
• 31:22are the things that drive
• 31:23people into treatment that that
• 31:24they wanna get back the
• 31:25part of themselves they wanna
• 31:26regain
• 31:27from entering recovery,
• 31:29and we don't measure it
• 31:30in our pharmaco studies at
• 31:32all. And I think that
• 31:33a lot of that is
• 31:35driven by them being difficult
• 31:36to measure metrics, And I
• 31:38think we continue to kinda
• 31:39do this. And so
• 31:40part of,
• 31:42part of this talk is
• 31:44to to to bring my
• 31:45own awareness to it, but
• 31:46also to to begin to
• 31:48ask questions about how we
• 31:48can do better with this
• 31:49and how continue to think
• 31:50about it. And I don't
• 31:51think that this is unique
• 31:53to to looking at,
• 31:54these outcomes. I don't think
• 31:55it's unique to addiction, even
• 31:57though I I put that
• 31:58in the main title slide.
• 31:59I think these are the
• 32:00same problems that are coming
• 32:01up in a lot of
• 32:01our research,
• 32:03where we tend to look
• 32:04most at the high risk
• 32:06individuals
• 32:07and,
• 32:08and and miss the population,
• 32:09and we continue to
• 32:11to look at what's easiest
• 32:12to measure, what's available, and
• 32:13and kind of perpetuate that.
• 32:16And so
• 32:18I think, as I said,
• 32:19again, I think this is
• 32:19something that's very common in
• 32:21addiction science.
• 32:23And as I reviewed,
• 32:24there isn't just really one
• 32:26reason for this.
• 32:27It's usually the confluence of
• 32:28factors. I think it's all
• 32:29of those things. I think
• 32:30that,
• 32:31usually it involves not interacting
• 32:32with the population enough, not
• 32:34understanding the individuals that are
• 32:36part of that population and
• 32:37what's going on in their
• 32:38lives is a big piece
• 32:39of it. I think the
• 32:41the idea of looking at
• 32:42sick individuals and focusing on
• 32:44that high risk group and
• 32:45othering them and saying they're
• 32:46not part of our population,
• 32:48is another piece of it.
• 32:49And then again, I think
• 32:50the metrics.
• 32:51And I think one of
• 32:51the things that came up
• 32:52when I was putting this
• 32:53all together is that we
• 32:54have just done this again.
• 32:56I don't know how many
• 32:56of you guys are familiar
• 32:57with kratom.
• 32:58It's a substance that we've
• 32:59been talking about a lot
• 33:00more.
• 33:01It's an it acts on
• 33:03opioids,
• 33:03receptors,
• 33:04same way opioids does. It's
• 33:06available,
• 33:07at a lot of, like,
• 33:08vape shops and stuff like
• 33:09that. You can become very
• 33:11dependent on them. And two
• 33:12weeks ago, Connecticut
• 33:14just rescheduled it and just
• 33:16dropped them from the shelves.
• 33:18But I don't I haven't
• 33:19seen any discussion of what
• 33:21to do when people are
• 33:22dependent on it. Maybe providers
• 33:23have received something
• 33:25that has said what to
• 33:26do, that there's evidence that
• 33:27buprenorphine can work for individuals
• 33:28that are,
• 33:29dependent on it. I have
• 33:31not seen that discussion occurring.
• 33:33I haven't the discussion recurring
• 33:34of where to get help
• 33:35or anything else, and I
• 33:36think it's the same kind
• 33:37of thing where you fail
• 33:38to understand the complexity of
• 33:40it. We kind of look
• 33:40at one aspect, which is
• 33:41supply over and over again,
• 33:43and kind of pulling out
• 33:44supply without thinking of the
• 33:45individual and what they will
• 33:47do next.
• 33:49I sent a text before
• 33:50this to try to talk
• 33:51with some research to see
• 33:53if we can learn something
• 33:54if it's not too late
• 33:55about what those individuals are
• 33:56doing. I think it's less
• 33:58likely they'll go to an
• 33:58illicit supply, like, with, with,
• 34:01like, OxyContin or something else,
• 34:03but,
• 34:03I think that could still
• 34:04happen. Just Yeah. Yeah. I
• 34:07like that. This is it
• 34:08is real, like, even before
• 34:09Kratom was rescheduled.
• 34:11Clinically, we see we see
• 34:12this people who, like, suddenly
• 34:13stop Kratom, and they don't
• 34:14know why they feel like,
• 34:15there's a real mystery because
• 34:16they just don't know why
• 34:17they feel so horrible. Yeah.
• 34:19So we are we are
• 34:20using buprenorphine. I can think
• 34:21of, like, a handful of
• 34:21patients. But you're right. Like,
• 34:22I don't know. It's just
• 34:24so unknown. Like, someone's like,
• 34:25I can't get anymore, and
• 34:26then they just feel horrible,
• 34:27and they just don't know
• 34:28why. Yeah.
• 34:30It's a confounding
• 34:31a confounding thing to to
• 34:32deal with. So it does
• 34:33happen. I I haven't thought
• 34:34of people trying to surveil
• 34:36Connecticut and sort of if
• 34:37there'll be an uptick. If
• 34:38people suddenly have a withdrawal
• 34:39symptoms, they're not knowing why
• 34:40and sort of trying to
• 34:41get into treatment. Yeah.
• 34:43Yeah. No. We'll see. It
• 34:44would have been best to
• 34:45start a study,
• 34:46you know, two months ago.
• 34:48But, you know, since that
• 34:49didn't happen as far as
• 34:50I know, it's still something
• 34:51that's worth doing because other
• 34:52states are gonna continue to
• 34:53do this too. So learning
• 34:54from these kind of situations
• 34:55is exactly the kind of
• 34:56things I like to do.
• 34:58So, for those who are
• 35:00interested,
• 35:02this this text,
• 35:03system science and population health
• 35:05is a great book. It's
• 35:06edited by Abdul El Sayed
• 35:07and Sandro,
• 35:08Gala.
• 35:10The the topic of this
• 35:11talk, wrong answers, I wrote
• 35:13a chapter on it. It's
• 35:14not specific to addiction policy.
• 35:15There's another one that I
• 35:16I authored on, social determinants
• 35:18of health and how system
• 35:19science can help with that.
• 35:20But it's a really great
• 35:21text. And for those who
• 35:22are not familiar, the first
• 35:24editor on there, Abdul El
• 35:25Sayed, actually stepped away from
• 35:26academia, and he's now running
• 35:28for US Senate of Michigan
• 35:30to change policy directly.
• 35:32So it'll be interesting to
• 35:33see what an epidemiologist
• 35:34comes up with there.
• 35:36And so I I appreciate
• 35:38this. My my goal is
• 35:39to kind of start discussions
• 35:40and and continue to think
• 35:41about how this affects,
• 35:43my work and hopefully others
• 35:44work.
• 35:46So I can't tell you
• 35:47how much of a privilege
• 35:48it is to be here
• 35:49and to be able to
• 35:49give this talk and to
• 35:51be part of this community.
• 35:53It's been very nice. And
• 35:54so I appreciate the opportunity
• 35:56to, to talk with you
• 35:56about my research and kind
• 35:58of my path here today.
• 35:59So happy to continue that
• 36:07discussion.
• 36:08Yes. David, thank you. Really
• 36:10nice talk. A lot of
• 36:11provocative
• 36:12themes and ideas,
• 36:14which I think this forum
• 36:15is really well suited for.
• 36:16So I appreciate you,
• 36:18taking the time.
• 36:19I wanna focus on your
• 36:21last topic that I I
• 36:22would call, you know, how
• 36:24do we get better patient
• 36:25reported outcomes
• 36:28to, you know, things that
• 36:29matter to patients in their
• 36:31lives
• 36:32and use data to
• 36:35drive,
• 36:37decision making and interventions towards
• 36:39things that actually affect patients.
• 36:41So, you know, in the
• 36:43in the clinical trial world,
• 36:46PCORI and even the NIH
• 36:49to a certain degree have
• 36:49been really pushing us to
• 36:51find better patient use better
• 36:53patient reported outcomes.
• 36:56But trials only go so
• 36:58far. Right? It takes a
• 36:59long time to develop a
• 37:01trial. We've got a certain
• 37:03select population of people who
• 37:04enter clinical trials, and it
• 37:06takes a long time to
• 37:07generate that kind of evidence.
• 37:08So
• 37:10what I and others have
• 37:11been thinking about is, you
• 37:12know, how can we get
• 37:14actual PROs into clinical practice
• 37:16so we're measuring
• 37:18things in day to day
• 37:19practice
• 37:20that actually matter to patients
• 37:22so that we can look
• 37:23at real time data and
• 37:25have that affect clinical outcomes.
• 37:26So
• 37:27all pointing to
• 37:29the VA is doing this
• 37:31in in the pain world,
• 37:33integrating
• 37:34a pain measure order set
• 37:37so that, you know, we
• 37:39can collect real time data
• 37:41and do secondary analysis of
• 37:43these data to better inform
• 37:44code of practice.
• 37:45And I just wanted to
• 37:47tell you that so that
• 37:48maybe you'd be interested in
• 37:49joining us to That that's
• 37:51fair. Help develop that measures
• 37:53and help examine the findings
• 37:55that we can
• 37:56create from this work. Yeah.
• 37:58No. I definitely am. I
• 38:00I I think it's so
• 38:01important.
• 38:02I I've been involved with
• 38:04some of the NIDA CTN
• 38:05studies. Right? And so, Ned
• 38:07Nunez,
• 38:08who I I collaborate with
• 38:10some,
• 38:10he we we worked on
• 38:12one of his, the Xspot
• 38:14study, which is looking at
• 38:15extended release, buprenorphine and naltrexone
• 38:18comparison. But they had such
• 38:19a nice question in there
• 38:20that the each patient reported
• 38:22what their goal was. What
• 38:23was your goal in entering
• 38:24treatment? And then you could
• 38:25look at, did that goal
• 38:26achieve? And we can't do
• 38:27that in these pharmacoepi studies
• 38:29if we don't have those
• 38:30kind of metrics,
• 38:32and we just don't have
• 38:33them. So, unfortunately, I don't
• 38:35no solution that I have
• 38:36yet except for things like
• 38:37this where the VA system
• 38:38can care about it and
• 38:39then enter it into the
• 38:40system.
• 38:41And we have Question one
• 38:43of the pain measure order
• 38:44status. What is your goal?
• 38:45Yes. Such a simple question.
• 38:48Yeah. And sometimes those questions
• 38:49are super important. I one
• 38:50of one of the papers
• 38:52I did a while ago,
• 38:53and it was it was
• 38:54one of those really fun
• 38:55papers in some sense. It
• 38:56was
• 38:57I shouldn't use that word.
• 38:58It was an interesting paper,
• 38:59but it was looking at,
• 39:00post deployment,
• 39:02responses that people had on
• 39:03on mental health and well-being.
• 39:05And so we collected all
• 39:06these measures on how people
• 39:07could be, and their post
• 39:09deployment and their their mental
• 39:10health status and their physical
• 39:11health status. And we asked
• 39:13one question that said, how
• 39:14is your post deployment
• 39:15transition going? And that was
• 39:17more predictive than anything else
• 39:18about all the other things
• 39:20and about how they thought
• 39:21it was. It was lower
• 39:22suicide attempt risk, all these
• 39:23other things. It was just
• 39:24a simple question. So sometimes
• 39:26that simple question can be
• 39:27added. It just needs to
• 39:28be added. And it's very
• 39:29difficult to change an entire
• 39:31system,
• 39:32to to to do that.
• 39:33But it's it's great when
• 39:34systems like the VA are
• 39:35willing to start.
• 39:39Yes.
• 39:40Great talk.
• 39:41I I would back just
• 39:43a little bit on your
• 39:44population
• 39:45figure because it all gets
• 39:47down to cost benefit analysis.
• 39:49Right? And in some cases,
• 39:51identifying
• 39:52this group is the most
• 39:53cost effective thing to do.
• 39:54Right? In other cases, not
• 39:56depending on the penetrance,
• 39:58depending on the cost of
• 39:59the intervention. Right? There's a
• 40:00lot of other Yeah. Go
• 40:01into that calculation.
• 40:03So I think it's a
• 40:04little bit dangerous to say
• 40:05it's always better. No. Yeah.
• 40:07You know, there are circumstances
• 40:08where it's better, and it
• 40:09sounds very much like this
• 40:10is one.
• 40:11But there are other circumstances
• 40:13where that would not be
• 40:14the case. And in fact,
• 40:15I would contend that we've
• 40:16gotten into a lot of
• 40:17overtreatment
• 40:18in this country
• 40:19precisely with that kind of
• 40:20logic. So
• 40:22I think it has to
• 40:23be a little more balanced
• 40:24than that. I appreciate that,
• 40:26and I
• 40:27take it honestly and say,
• 40:28yes. I can I will
• 40:30change how I present it
• 40:31because I don't feel that
• 40:31way? I'm not saying that
• 40:32I don't think the high
• 40:33risk approach is
• 40:34wrong and not useful,
• 40:36and I think that both
• 40:37have their place in society.
• 40:38I think the challenge that
• 40:39we have is too often
• 40:41biases
• 40:42is what draws to the
• 40:43high risk approach because what
• 40:45the high risk
• 40:46approach requires from everybody else
• 40:48is nothing, and that is
• 40:50much more tenable. And so
• 40:51especially when you're talking about
• 40:52a population or outcome that
• 40:53is,
• 40:55stigmatized, we're going to favor
• 40:57that one over the one
• 40:58that makes me change what
• 40:59I'm doing if I'm not
• 41:00part of that just to
• 41:01help them. And we saw
• 41:02that during COVID. Right?
• 41:04Yes.
• 41:05With alcohol, for example. Yes.
• 41:07I don't disagree with No.
• 41:08It's how I presented it.
• 41:09So I I hear that.
• 41:10Thank you.
• 41:12Yes.
• 41:13Online had a question.
• 41:16You wanna jump on, or
• 41:17I can read it?
• 41:20I don't know how. Hey,
• 41:22David. Can you hear me?
• 41:23Yes.
• 41:24Great talk.
• 41:26Appreciate,
• 41:29the narrative over time.
• 41:32You mentioned that one of
• 41:33the common outcomes that,
• 41:36studies were using you identified
• 41:38for buprenorphine was retention and
• 41:40treatment at a hundred and
• 41:41eighty days.
• 41:43And that sounds fairly straightforward.
• 41:45I was just
• 41:47my observation is that
• 41:49different teams operationalize
• 41:51even that, quote, unquote, standard
• 41:54metric differently.
• 41:55And as you pointed out,
• 41:56some will allow, you know,
• 41:58seven days of missed med,
• 41:59then some will
• 42:01expand it to thirty days.
• 42:02So I was just wondering
• 42:03if you could kinda talk
• 42:04on that
• 42:05variability.
• 42:07Yes. Yeah. Even even those
• 42:08measures are not completely agreed
• 42:09upon.
• 42:11I mean, I think the
• 42:12thirty day gap is the
• 42:13most commonly used one, but
• 42:15that does bring up the
• 42:16fact that by thirty days,
• 42:18every single person that's on
• 42:19buprenorphine of a reasonable dose
• 42:22would be deep into withdrawals.
• 42:24You know,
• 42:25after one day, two days,
• 42:26they would be into withdrawals.
• 42:27It's a longer active medication,
• 42:28I believe, but not not
• 42:30enough to do longer periods.
• 42:31So, yes. I I think
• 42:32it's interesting that that's the
• 42:34gap that's often used when
• 42:35we have to assume that
• 42:36people are using something else
• 42:38during that time probably.
• 42:39Yeah. Historically,
• 42:40I think that gap derives
• 42:42from payment for opioid treatment
• 42:44programs. Yes.
• 42:45So it's not clinically derived.
• 42:48Yes. No. A lot of
• 42:49I mean, that's that was
• 42:50the point I was kinda
• 42:51trying to make where that's
• 42:52the process outcome.
• 42:53And I think that's part
• 42:54of the process. It's really
• 42:55more about health services research
• 42:57that has dictated that.
• 42:58I think that thirty day
• 43:00gap
• 43:01I can't think of the
• 43:01organization it comes from. The
• 43:03n
• 43:04n n any anyway, it's
• 43:06a very common one. I
• 43:07either way, it is a
• 43:08process gap. It is based
• 43:09on payment. It's not based
• 43:10on, factual
• 43:12piece. So
• 43:13I think that that is
• 43:14an area we can improve
• 43:16as well as even the
• 43:16ones we already collect. How
• 43:18do we collect them in
• 43:19a more meaningful way that's
• 43:20thinking about the patient's
• 43:22responses,
• 43:23and and where they're at
• 43:24versus
• 43:25just how things are paid,
• 43:27I guess.
• 43:30I I have a,
• 43:32another question. So I this
• 43:34last sort of cuts up
• 43:35sort of when you went
• 43:35through that list of what
• 43:36your patients report as as
• 43:38their goals and sort of
• 43:39that bottom sort of being,
• 43:40like, employment or avoiding criminal
• 43:42justice,
• 43:43contacts, sort of reconnecting with
• 43:45family.
• 43:46I I'd love your thoughts
• 43:47on sort of like there
• 43:48are it's challenging,
• 43:50but there are ways to
• 43:51get at administrative data sets
• 43:53to get at some of
• 43:55and, like, I've spent a
• 43:56lot of time trying to
• 43:57get access to that data,
• 43:59through, like, you know, using
• 44:01IRS records or using employment
• 44:03records or using some or
• 44:04sort of criminal justice records.
• 44:06It's a challenge. But is
• 44:08that
• 44:09doing that type of work
• 44:10where you're linking sort of
• 44:11treatment to sort of other
• 44:12datasets,
• 44:13does that actually achieve what
• 44:15you would have in mind?
• 44:16Because, like, is it still,
• 44:17like, not
• 44:18accurate on the patient centered
• 44:19goals or sort of, like,
• 44:20still it's like me saying
• 44:22that optimal goal is
• 44:24seventy five percent employment or
• 44:25whatever whatever make up the
• 44:26number. Is that type of
• 44:28work sort of meet some
• 44:29of the needs, or is
• 44:30it still insufficient
• 44:31to do that?
• 44:33Yes. That makes sense.
• 44:35I think
• 44:37I don't have a clear
• 44:38answer to it. I think
• 44:39that that's more of, an
• 44:41area I'd like to continue
• 44:42to dive into, and I'm
• 44:43happy to talk more about
• 44:44it as well. I think
• 44:45that
• 44:47anything that you look at
• 44:48besides what is just this
• 44:50is what we do and
• 44:51what we've done is an
• 44:52improvement, because it starts to
• 44:53understand the complexity
• 44:55of the patient experience.
• 44:57And when we continue to
• 44:58just use the same metrics
• 44:59because that's what's been used
• 45:00before, I think we lose
• 45:02that.
• 45:03And again, I'm not unique
• 45:05to it. I I I
• 45:06am guilty of this as
• 45:07well sometimes where we tend
• 45:08to just go, this is
• 45:08what what's been measured. Let's
• 45:10stick with it. So I
• 45:11think anytime you try to
• 45:12advance that, it's good. But
• 45:13I think that that is
• 45:14empirically something that we could
• 45:16look at. And I think
• 45:17that you would do studies
• 45:18where you ask people if
• 45:20how their treatments are going,
• 45:21and you also pull in
• 45:22this data so you understand
• 45:24if people are getting jobs,
• 45:25are they feeling like the
• 45:26recovery is going well? I
• 45:27mean, it's plausible that getting
• 45:28employment could actually,
• 45:30you know, be negative to
• 45:31recovery, especially if it happens
• 45:33too soon or something else.
• 45:34I don't know. So I
• 45:35don't think it's as easy
• 45:36to say this is what
• 45:37we need to measure because
• 45:38that's gonna solve it. I
• 45:39think it's
• 45:40we need to do better
• 45:42with how we're thinking about
• 45:44capturing success,
• 45:45of these policies and these
• 45:47changes.
• 45:48Yeah. Along those lines, I
• 45:50would say we need to
• 45:50do better with patient reported
• 45:52outcomes. Yes. It's just a
• 45:53huge individual variability
• 45:55in those those values so
• 45:57that it can be
• 45:59fairly misleading.
• 46:00I years ago, I did
• 46:01a study where we were
• 46:02looking at quality of life,
• 46:03reported self reported quality of
• 46:05life, health report,
• 46:07in people with HIV.
• 46:09And the group that had
• 46:10the quote worst quality of
• 46:12life were white men who
• 46:13were relatively healthy.
• 46:16And the group that had
• 46:17the best quality of life
• 46:19were older black men,
• 46:20which were who were quite
• 46:22sick.
• 46:23It's all relative to your
• 46:24environment
• 46:25in terms of how you
• 46:26report those factors. So
• 46:28I I
• 46:29well, I think it's really
• 46:30important to ask people how
• 46:32they're doing and the kinds
• 46:33of questions that you're raising.
• 46:34There are all kinds of
• 46:36measurement problems
• 46:37just trying to use those
• 46:38as outcomes that we need
• 46:40to figure out how to
• 46:40solve. I mean, I'm not
• 46:41saying abandon them, but I'm
• 46:43saying
• 46:44be dubious of the current
• 46:45ones that we use because
• 46:46they they have a lot
• 46:47of problems too. Yeah. No.
• 46:49Everything's
• 46:49very imperfect here. And I
• 46:51think in that example, you
• 46:52give perfectly, like, hopelessness is
• 46:54one of the number one
• 46:54predictors of suicide. Right? And
• 46:56so even if that person's
• 46:57in a
• 46:58a good social environment, everything
• 47:00looks good, but they are
• 47:01feeling hopeless, you know, they
• 47:03might be at the highest
• 47:03risk for an adverse event,
• 47:05whereas, you know, the other
• 47:06individual who's at a worse
• 47:07circumstance. So I I think
• 47:08it's very challenging, and I
• 47:10do appreciate the things like
• 47:11z codes exist, and maybe
• 47:12that's a piece of it.
• 47:13I know clinical notes exist.
• 47:15I don't know. I don't
• 47:16know. I mean, I think
• 47:17that that's why I'm presenting
• 47:18this. This is my this
• 47:19is where my head is
• 47:19most days now, is understanding
• 47:21how to measure these things
• 47:22better.
• 47:23And so I I was
• 47:24hoping to bring them up
• 47:24so other people would that
• 47:25are interested and it can
• 47:26continue this discussion.
• 47:28I think if you ask
• 47:29the same person
• 47:31Yeah. What's been in the
• 47:32series and they become their
• 47:33own control Yes. Is a
• 47:35more useful metric. But when
• 47:37you try to do it
• 47:37on a population level,
• 47:39you get into all kinds
• 47:40of problems. Yeah. I think
• 47:41that makes sense.
• 47:44Thank you for your talk.
• 47:47I'm just looking at the
• 47:48first author on the book
• 47:49that you're describing and thinking
• 47:50about. Are there communities or
• 47:52states
• 47:53in the United States where
• 47:55where simple, you know, interpretations
• 47:57have not created
• 47:58where where they're less inclined
• 48:00where they're where they're less
• 48:01reactive. Right? In theory, Connecticut
• 48:03in theory is is a
• 48:04pretty
• 48:05felt to be reasonable, generous
• 48:07state in terms of thinking
• 48:08about Medicaid.
• 48:10But, are there other places
• 48:11where where they've been more
• 48:14careful with data in terms
• 48:15of implementing both
• 48:18laws or guidelines,
• 48:21that you can share with
• 48:22us?
• 48:24I'm unfortunately not the right
• 48:25person to ask about that.
• 48:26I love the question, and
• 48:28I think it's something that
• 48:29I'd be very interested in
• 48:30knowing. I don't think I
• 48:31have enough
• 48:33firsthand experience with the policy
• 48:35making side of it. I
• 48:36mean, I've definitely I saw
• 48:38it.
• 48:39I worked for the US
• 48:40army for a bit in
• 48:41the public health command, and
• 48:42so we would get called
• 48:43out to do outbreak investigations
• 48:45of mental health issues by
• 48:47generals.
• 48:48And so you saw the
• 48:49different scope of generals where
• 48:51some of them would hear
• 48:51the data and just go,
• 48:52this is what I'm doing.
• 48:53I don't really care what
• 48:54you say. You'd see other
• 48:55people that want to have
• 48:56discussion with you. So that
• 48:58exists on a continuum, and
• 48:59I think it I'm sure
• 49:00it exists on a continuum
• 49:01elsewhere. I don't have examples
• 49:03of it. The exemplars. Right?
• 49:04So where where are people
• 49:05doing it? It doesn't have
• 49:06to be a cookie cutter
• 49:07at one size, but it's
• 49:08all it's gonna be very
• 49:09good. Yeah. Words.
• 49:11We know them.
• 49:12I am too.
• 49:13I wanna think of it.
• 49:17Yeah.
• 49:18Comment or the question comment?
• 49:20It's very hard to get
• 49:21people to fill out.
• 49:25One of the problems with
• 49:26the metric.
• 49:28The how what you're asking
• 49:29them to do,
• 49:31what they do. In the
• 49:32cancer field, roughly, I don't
• 49:33know, seven or eight years
• 49:34ago, a huge study came
• 49:35out, randomized.
• 49:37People were getting chemo.
• 49:39Half of them were randomized
• 49:40to patient
• 49:42electronic, and the
• 49:44alert would go off if
• 49:45they're having symptoms and the
• 49:46other half is usual care.
• 49:49People randomized to the PRO
• 49:52group ended up having
• 49:54actually better survival
• 49:55because the, the bell would
• 49:57go off if they're nauseous,
• 49:58and the nurse would call
• 49:59them and say, here you're
• 50:00nauseous. Let's give you this.
• 50:01We'll keep you out of
• 50:02the ER. They could stay
• 50:03on their treatment longer. Mhmm.
• 50:05If we were able to
• 50:06finish the course of therapy.
• 50:07The people came back from
• 50:08the big ASCO annual meeting.
• 50:10So this is a plenary
• 50:11presentation, and the person did
• 50:13a great job presenting
• 50:14to you. Basically, putting up
• 50:15their PRO, have the intervention,
• 50:18comparing it to all the
• 50:19big chemo drugs. Like, this
• 50:20is just as good as
• 50:22Beviz is and that. Then
• 50:24checkpoint,
• 50:25it sounds a lot cheaper.
• 50:26So everybody came back across
• 50:27the country. We've gotta do
• 50:29this. The of course, this
• 50:30is we gotta start measuring
• 50:31PROs and our time.
• 50:33Five years later, seven years
• 50:34later, almost nobody's doing. It's
• 50:36just real big so a
• 50:37big challenge
• 50:38is to figure out not
• 50:40only what to measure, when,
• 50:42where, and how to use
• 50:42it, but,
• 50:44how to make sure that
• 50:46we're
• 50:47eventually,
• 50:49getting clinics
• 50:50getting this embedded into the
• 50:51clinical
• 50:52workflow. Mhmm. Be one of
• 50:54the
• 50:55main people I've found helpful
• 50:56is if patients can see
• 50:58it helpful for them
• 51:00in some way. So I'm
• 51:00just thinking
• 51:02about how to improve
• 51:03it here. It's what we
• 51:04figure out what we're trying
• 51:05to do. Yeah. No. I
• 51:07I find that that's a
• 51:08interesting area that I haven't
• 51:10done as much in. I
• 51:11with prescription drug monitoring programs,
• 51:12we did a little bit
• 51:13in how the clinical workflow
• 51:14and understanding how that, like,
• 51:16enters into it.
• 51:18It's, that's a whole other
• 51:20area of study that I
• 51:20think I find interesting, but
• 51:22I am not as familiar
• 51:22with. So I hope I
• 51:23can learn more about that.
• 51:25Question was, just as far
• 51:27as, interventions, just thinking about
• 51:29this PRO Yeah.
• 51:30People's goals and
• 51:35what we want to.
• 51:36Are there inter studies ongoing
• 51:38or,
• 51:40interventions that are being evaluated
• 51:42that are basically
• 51:44multi prompt or adjuncts to
• 51:49traditional pharmaceutical
• 51:51approach
• 51:52that would be
• 51:53that have been shown to
• 51:54be effective in some way
• 51:56as far as, you know,
• 51:58having getting jobs, take avoiding
• 52:00conservation and things like that.
• 52:02I mean,
• 52:04yes and no. I I
• 52:05think that there are clinical
• 52:06trials that are collecting those,
• 52:07but that's where it's really
• 52:08at is the clinical trials.
• 52:09I haven't seen it outside
• 52:10of that. So again, this
• 52:12those CTN studies that are
• 52:13being done through the NIH
• 52:15of the clinical trials network,
• 52:16like, they definitely have those
• 52:17outcomes and they have published
• 52:18papers that have looked at
• 52:19those kind of things for
• 52:21buprenorphine and naltrexone, these other
• 52:22drugs.
• 52:24Haven't seen it outside of
• 52:25a clinical trial.
• 52:29But it's a discussion. I
• 52:30mean, editorials are everywhere now
• 52:31on this. So let's let's
• 52:33you know, this this came
• 52:35to my attention a few
• 52:35years back, and now it's
• 52:36like, I I feel like
• 52:38it's a flood, you know.
• 52:39I I think the scientific
• 52:40consensus kinda moves, and I'm
• 52:41not unique to it. So
• 52:43I think it's discussions that
• 52:43are happening. It's just nobody
• 52:45knows how to do it.
• 52:47Well, I have one more
• 52:47question on the Zoom. Is
• 52:49there okay.
• 52:51Good. One. Three question.
• 52:53Julia, do you wanna jump
• 52:55on?
• 52:57Hi. Sure.
• 52:59Thank you so much for
• 53:00sharing this fascinating work. So
• 53:01I'm thinking about the global
• 53:02perspective,
• 53:03making it even more complex.
• 53:05And,
• 53:06of course,
• 53:08we want to be able
• 53:09to compare what happens
• 53:10in different contexts
• 53:12and learn from that.
• 53:14But then this,
• 53:15question that you are,
• 53:18grappling with and showing us
• 53:20becomes even more complicated because
• 53:22validating
• 53:23even just one instrument for
• 53:24different countries takes a lot
• 53:26of effort and a lot
• 53:27of work.
• 53:28And then,
• 53:29if you validate it for
• 53:31the language and then, for
• 53:32example, like, if it's the
• 53:34language in England versus the
• 53:35language in Ireland, this is
• 53:37also,
• 53:38an extra validation process. So
• 53:41is it, you know, is
• 53:42the juice worth the squeeze?
• 53:44Do you think that
• 53:45this is effort wisely spent?
• 53:48What would be your recommendation
• 53:50given what you've been learning?
• 53:52Yes.
• 53:53That's a a great point.
• 53:59Is it worth it? I
• 54:01mean,
• 54:02I think it is, but
• 54:03I I think I also
• 54:04understand that
• 54:06it's going to have translational
• 54:08problems.
• 54:09And so then it needs
• 54:10to be studied again and
• 54:11again in different contexts. So
• 54:14I think that the one
• 54:15of the questions that will
• 54:16have to come up is
• 54:17what is the consequence of
• 54:18using one measure over another?
• 54:20And I don't think we
• 54:21have that yet.
• 54:22And so maybe
• 54:24there have been studies that
• 54:25looked at hundred and eighty
• 54:26days and found it to
• 54:27be associated I'm I'm just
• 54:29using this example. I found
• 54:31it to be associated with
• 54:33reductions and, overdose deaths, and
• 54:35that's probably the main outcome
• 54:37they probably looked at. But
• 54:39it's not a perfect relationship.
• 54:41And so like many
• 54:43things, there's a lot
• 54:44left that's not understood. And
• 54:46so, like, you know, if
• 54:47it reduces if it's a
• 54:49two percent reduction in overdoses,
• 54:51that could be scientifically significant,
• 54:53and could be something that
• 54:53we continue to use for
• 54:54that reason.
• 54:56But is it meaningful? And
• 54:57I I think that's a
• 54:58discussion that we have to
• 54:59continue to have. I I
• 55:00don't have an answer for
• 55:01it.
• 55:04I I I hope that
• 55:05next time I present, I
• 55:06will have more thoughts as
• 55:07I kinda dive into this
• 55:08research more empirically and understand
• 55:10that. But I I think
• 55:10that's a great point of
• 55:12understanding how much are we
• 55:13gaining from these different measures.
• 55:15It's one thing to to
• 55:17encourage researchers and policymakers to
• 55:19think about the whole
• 55:21individual instead of just a
• 55:22piece of it. It's another
• 55:23to say what that costs,
• 55:25and that's not something I
• 55:26do as much. So I
• 55:28appreciate
• 55:29being reminded of my own
• 55:31limitations,
• 55:33in that piece. So, yeah,
• 55:34I think that's an interesting
• 55:35point to continue to think
• 55:36about.
• 55:38Well, it's a great session.
• 55:39Thank you. Thank you.