INFRASTRUCTURE AND APPLICATIONS IN IMAGE-GUIDED RADIATION ONCOLOGY BREAST AND PROSTATE IMAGING

Transcript

• 00:00Okay. So we're gonna switch
• 00:01gears and talk about image
• 00:02analysis and AI. And, incidentally,
• 00:04if you sign up, you're
• 00:05gonna get information about the
• 00:06AI interest group in the
• 00:07institute.
• 00:08We'll do some office hours
• 00:09and some training seminars for
• 00:10that. So we have three
• 00:11speakers. Doctor. Guarim is first,
• 00:13who's an assistant professor from
• 00:15France,
• 00:15moved to Yale recently, and
• 00:17he works in both pet
• 00:18reconstruction
• 00:19and algorithms for head
• 00:21imaging. And then we also
• 00:23have doctor John Onofrey, who's
• 00:25also in some processing radiology
• 00:27and joined with urology, and
• 00:29he works in variety of
• 00:31outcome development, particularly prostate cancer,
• 00:33and then doctor Nizhevornick who's
• 00:34medicine for some radiology.
• 00:37And they both have the
• 00:38Hopkins Yale kind of pipeline
• 00:41here, and
• 00:42work is also in outcome
• 00:43development and also
• 00:45brain diseases and more recently
• 00:46in some of the predicting,
• 00:50birth issues with,
• 00:52accreta. I will not try
• 00:53to name that. So with
• 00:54that,
• 01:01Thank you for the introduction.
• 01:03My name is Thibaut Marine.
• 01:04I'm an assistant professor in
• 01:05the department of radiology and
• 01:06biomedical imaging with a secondary
• 01:08appointment in BITS, and I'm
• 01:09with the Yale Biomedical Imaging
• 01:11Institute. Today, I'm gonna talk
• 01:12about data science infrastructure for
• 01:14it and some example of
• 01:16processing pipelines that we have
• 01:18established.
• 01:19So, as we've seen through
• 01:20the description of the calls
• 01:21and the imaging modalities available
• 01:23at Yale, we are in
• 01:24a unique position to use
• 01:26data from a lot of
• 01:27modalities, different research groups.
• 01:29What we are trying to
• 01:30build is bridges so that
• 01:31that can become cross modality
• 01:33and across group. We have
• 01:34data from PET, from MR,
• 01:35from the research calls. We
• 01:37also have data from the
• 01:37hospital.
• 01:39It's across the street, but
• 01:40that data is more difficult
• 01:41to access because of safety
• 01:43concerns and privacy concerns. So
• 01:45what we are trying to
• 01:46establish is a centralized repository
• 01:49for data first.
• 01:50In that central repository, data
• 01:52from the calls will be
• 01:53collected. Data from the hospital
• 01:54will also have a a
• 01:55bridge that is, allowing us
• 01:57to get that data in
• 01:58a secure way and host
• 01:59it on servers where they
• 02:01can be hosted safely
• 02:02without,
• 02:03identification information.
• 02:05Along with that, data repository,
• 02:07we need it to be
• 02:08searchable. So we need a
• 02:09database and, an index, and
• 02:11we need computing, power because
• 02:13all the processing pipeline pipelines
• 02:15we've looked at through the,
• 02:16prism of applications,
• 02:18they require data processing and
• 02:20so they require resources.
• 02:21To this end, we're renovating,
• 02:23rooms in the,
• 02:25one hundred Church Street,
• 02:28building
• 02:29to host those servers. So
• 02:30this has been a long
• 02:31process,
• 02:32but over the last few
• 02:33months, the room has, gotten
• 02:35ready, and we now have
• 02:36a picture of the room
• 02:37that is ready for use.
• 02:38This room will host a
• 02:40combination of,
• 02:41computing power and hosting
• 02:43for data storage.
• 02:45And so CPUs, GPUs that
• 02:47are able to work with
• 02:48the large amount of data
• 02:49we're dealing with. Imaging is
• 02:51pretty large compared to other
• 02:52data processing pipelines, so we
• 02:54need sufficient storage
• 02:55along with sufficient computing power.
• 02:57And so that's what we
• 02:58are working to build.
• 03:00The data that comes from
• 03:02the hospital comes de identified.
• 03:04So in parallel, led by
• 03:05Zinnios, we're looking at a
• 03:07pipeline where we can get
• 03:08data from the hospital de
• 03:10identified.
• 03:11So we have this tool
• 03:12that comes from, from Duke
• 03:13that, is the identification tool
• 03:15that removes
• 03:16all the HIPAA data and,
• 03:18has expert determination, which means
• 03:20that the data can be,
• 03:22transferred to us. Our servers
• 03:23also,
• 03:24comply to the safest procedures
• 03:26and, guidelines for hosting,
• 03:28patient data and subject data.
• 03:30And we are building that
• 03:31framework so this data can
• 03:33be combined with the processing
• 03:34units that we have so
• 03:35that we can make great
• 03:37science together and not have,
• 03:39red tape along the way,
• 03:41but still having the safety
• 03:42of the data. So once
• 03:43we collect the data, we
• 03:44need to make it accessible
• 03:45and searchable. And to this
• 03:47end, we are building a
• 03:48cross modality database that will
• 03:50list all the existing data
• 03:51set that we have,
• 03:53from the different imaging calls.
• 03:55So that people can look
• 03:56and search data that can
• 03:57be useful for their research,
• 03:58for comparisons.
• 03:59Or as George mentioned,
• 04:01when building a new study,
• 04:02maybe very valuable to realize
• 04:04that the control group is
• 04:05already available that can save
• 04:06imaging cost quite a bit.
• 04:08On the table here is
• 04:10shown the list of, PET
• 04:11scans that we've had and
• 04:13sorted by traces.
• 04:15On the PET side, we
• 04:16already have a great database
• 04:17that collects all of this
• 04:18since two thousand five with
• 04:19more than twenty thousand scans.
• 04:21We are building,
• 04:22something that will include MR
• 04:24and PET and be able
• 04:25to be cross searched.
• 04:28Moving away from,
• 04:30data infrastructure to data processing,
• 04:34I wanna talk about a
• 04:35couple of projects we're working
• 04:36in that end, and they
• 04:37are both related to cancer
• 04:39and treatment planning. So the
• 04:41first one is,
• 04:42we're trying to develop methods
• 04:44that can assist radiologists and
• 04:46radiation oncologists for sarcomas and
• 04:48head and neck tumors. The
• 04:49process
• 04:50typically, is after imaging,
• 04:53doing some contouring of the
• 04:55gross tumor volume, the GTV,
• 04:56followed by the clinical target
• 04:58volume, which will be what
• 05:00the radiation will target.
• 05:02And so we're trying to
• 05:03build a pipeline where we
• 05:04have AI all along the
• 05:06way. We have basically two
• 05:07goals. The first one is
• 05:08end to end pipeline where
• 05:09AI can assist radiologists and
• 05:10radiation oncologists.
• 05:12And the second goal is
• 05:13to incorporate variability in the
• 05:15models.
• 05:16One danger of AI is
• 05:17when it gives an answer
• 05:18and we trust this without
• 05:19questioning it. We're trying to
• 05:21develop models that will have,
• 05:23an idea of the confidence
• 05:24they are providing in the
• 05:25images they are giving us.
• 05:27This way, the human is
• 05:28always in the loop and
• 05:29we can have a better
• 05:31targeted
• 05:32analysis from the humans.
• 05:34That variability is something we
• 05:35have quantified,
• 05:37inter and intra reader for
• 05:38soft tissue sarcomas, and we
• 05:40observe that there is always
• 05:41an amount of variability that
• 05:43is, that is present there
• 05:45within readers and across readers.
• 05:46So we quantified this, and
• 05:48we developed methods that allow
• 05:50us to utilize this in
• 05:51the model rather than fight
• 05:52it, basically. It's actually valuable
• 05:54information. The key observation is
• 05:56that,
• 05:57variability across readers is indication
• 06:00of confidence. When people are
• 06:01confident, the variability will be
• 06:02lower, and that can be
• 06:03very valuable information. If we
• 06:05can provide segmentations to readers
• 06:08that have that notion of
• 06:10confidence, they can spend the
• 06:11time on the part of
• 06:12the image that makes sense.
• 06:13So that will improve the
• 06:14throughput.
• 06:16This slide shows, some of
• 06:18our results and, and papers
• 06:20on this work where we
• 06:21use diffusion models to, predict
• 06:23confidence map, and we, show
• 06:25that we have,
• 06:26we tend to outperform existing
• 06:28state of the art methods
• 06:29using those advanced AI tools.
• 06:31We're also extending this work
• 06:33to head and neck tumors,
• 06:34where we're also developing new
• 06:36models that combine diffusions
• 06:37and,
• 06:38vision transformers,
• 06:40typical models using generative
• 06:43AI to improve the accuracy
• 06:44and the precision of segmentations.
• 06:47The second project I wanna
• 06:48touch on, is, prostate imaging
• 06:50where we're also trying to
• 06:51look at diagnostics
• 06:53and how AI can assist
• 06:54that. So as it was
• 06:56discussed in a previous talk,
• 06:57diagnostics will
• 06:59be able to, have injection
• 07:01of lutetium that will be
• 07:02imaged through the process. For
• 07:03the next few days, we
• 07:05can image
• 07:07we use inspect the lutetium,
• 07:08and we can build a
• 07:09dose map.
• 07:11This is a well known
• 07:12task how to estimate the
• 07:13dose from the spec. What
• 07:14we are trying to achieve
• 07:15is
• 07:16where we need the the
• 07:17AI tools. We're trying to
• 07:19see whether from the pet,
• 07:20we can get information on
• 07:21those. That's one. And second,
• 07:24from that dose, can we
• 07:25predict outcomes? Can we refine
• 07:26treatment? Can we personalize the
• 07:28treatment? And this is why,
• 07:30this is particularly challenging. It's
• 07:32challenging because the PET and
• 07:33the dose don't correlate the
• 07:35same way in different organs.
• 07:37So but it's very important
• 07:38for us to do it
• 07:39from the PET. One intermediate
• 07:41step we're trying to do
• 07:42is, well, from a single
• 07:43SPECT trying to get the
• 07:44dose. The reason why this
• 07:46is important to do from
• 07:47the PET is as we
• 07:48move from Lutetium to Actinium,
• 07:49SPECT is no longer an
• 07:50option. And so we PET
• 07:52is the only thing we
• 07:53can have to, assess the
• 07:54dose as we go.
• 07:56And that's why we're building
• 07:58this holistic framework where instead
• 08:00of having a treatment which
• 08:01is set once and for
• 08:02all and doesn't change, we're
• 08:03trying to use AI everywhere
• 08:05along the way to adjust,
• 08:07predict the outcomes, and refine
• 08:09the the planning, if anything,
• 08:10to know when to stop
• 08:11when treatment is working.
• 08:14That concludes my talk on
• 08:16this, and I will give
• 08:17the microphone to jump on
• 08:19the floor.
• 08:25Thank you. I am going
• 08:26to now talk for give
• 08:28you about the six minute
• 08:29overview
• 08:30of prostate cancer image analysis
• 08:32here at Yale.
• 08:34So prostate cancer is a
• 08:36global problem.
• 08:37In the United States alone,
• 08:39approximately eleven percent of males
• 08:41will receive a prostate cancer
• 08:43diagnosis over the course of
• 08:44their lives.
• 08:46This means that prostate cancer
• 08:47accounts for for approximately thirty
• 08:49percent of cancers in men,
• 08:51which is estimated to affect
• 08:52more than three hundred thousand
• 08:54and results in greater than
• 08:55thirty five thousand deaths per
• 08:57year.
• 08:59So imaging is a critical
• 09:01component found throughout the prostate
• 09:03cancer
• 09:04diagnostic pipeline.
• 09:05So after initial screening with
• 09:07prostate specific antigen blood testing,
• 09:10radiologists
• 09:11then interpret and read multi
• 09:13parametric MRI to identify suspected
• 09:16lesions according to the PI
• 09:17RADS reporting standard.
• 09:19Urologists then use ultrasound to
• 09:21perform image guided biopsy.
• 09:23Pathologists
• 09:24then interpret this biopsy tissue,
• 09:26which can be digitized using
• 09:28whole slide imaging.
• 09:29And then finally, the patient
• 09:30can be stratified into risk
• 09:32groups to determine whether they
• 09:33receive active surveillance
• 09:35or they go on to
• 09:36immediate treatment.
• 09:37Now imaging continues to be
• 09:39important during the treatment process.
• 09:41For example,
• 09:42PSMA PET and SPECT can
• 09:44be used for theranostics,
• 09:46as we've seen, and CT
• 09:47imaging can be used for
• 09:48radiation therapy.
• 09:51Image analysis powered by artificial
• 09:53intelligence and machine learning methods
• 09:56seeks to leverage,
• 09:58all these imaging modalities to
• 10:00enhance clinical insights. And this
• 10:02is accomplished through a few
• 10:04tasks such as classification,
• 10:06segmentation,
• 10:07registration, and data integration.
• 10:10So one of the first
• 10:10fundamental tasks in prostate image
• 10:12analysis is to segment the
• 10:14gland within MRI.
• 10:15Not only do these volume
• 10:16measurements,
• 10:17are they used to allow
• 10:18allow us to compute PSA
• 10:19density,
• 10:20but it also serves as
• 10:21a preprocessing step for many
• 10:23of the analyses that we'll
• 10:24see following here. However, this
• 10:26is a challenge due to
• 10:27the variability, the heterogeneity, the
• 10:29MR images themselves that you
• 10:30see in this picture. And
• 10:32additionally, these AI algorithms can
• 10:34are overly sensitive, can over
• 10:35train to the scanner themselves.
• 10:37So as demonstrated by techniques
• 10:40developed here at Yale, image
• 10:41analysis is critical to robustly
• 10:43apply these AI algorithms across
• 10:45different clinical sites.
• 10:47And note for all of
• 10:48you AI users,
• 10:50don't ever trust an algorithm
• 10:51unless you validate against, like,
• 10:53multi site data or external
• 10:54data. K?
• 10:56So going beyond gland segmentation,
• 10:58we also wanna differentiate different
• 11:00zonal anatomy of the prostate,
• 11:01which is part of their
• 11:02PyRAADS reporting, system.
• 11:04In this work here, we
• 11:05directly integrated both anatomical and
• 11:08geometric constraints to segment this
• 11:10challenging boundary.
• 11:12This approach has been directly
• 11:14translated,
• 11:15into clinical use
• 11:17by our commercial partners, Eigen
• 11:19Health, into their, ProFuseCAD
• 11:21software platform. So this is
• 11:23pretty exciting to see it
• 11:24actually going from the lab
• 11:25to the real world.
• 11:27And our final MRI segmentation
• 11:29application here is to identify
• 11:31lesions,
• 11:32which are extremely challenging to
• 11:34identify even for experienced radiologists.
• 11:36So to tackle this problem
• 11:38here, we developed an approach
• 11:39that directly integrates clinical data,
• 11:42in this case, PSA,
• 11:43into the segmentation
• 11:45process.
• 11:46So this human in the
• 11:47loop approach allows the radiate
• 11:49radiologist then to mimic the
• 11:50variability in PSA levels
• 11:53and is able and you're
• 11:54able to visualize the effect
• 11:55it has on the algorithm
• 11:56segmentation
• 11:57output. So from this image
• 11:58here you can see from
• 11:59left to right as you
• 12:01increase the PSA value
• 12:03the lesion actually grows during
• 12:05the segmentation process.
• 12:07This too is being integrated
• 12:09into the ProFuseCAD system, so
• 12:11another good example of translational
• 12:13science here.
• 12:14We can also perform classification
• 12:16using the MRI to predict
• 12:18if patients have benign, non
• 12:19significant disease, or clinically significant
• 12:21prostate cancer. So as shown
• 12:23here, using the image by
• 12:24itself is quite underwhelming in
• 12:26this AUC curve.
• 12:27However, by integrating additional clinical
• 12:30data such as patient age,
• 12:31PSA, and PI RADS reporting,
• 12:33we see the synergistic effect
• 12:35where that significantly enhances the
• 12:37prostate cancer classification
• 12:38performance.
• 12:40Next, during the biopsy procedure,
• 12:43lesions are extremely challenging to
• 12:44identify in the ultrasound image.
• 12:46Therefore, what we want to
• 12:47use is MRI to identify
• 12:49lesions of interest for targeted
• 12:51biopsy.
• 12:52The challenge here is coregistration
• 12:53or aligning these two vastly
• 12:55different imaging modalities.
• 12:56So to overcome this hurdle,
• 12:58we use a surface based
• 12:59registration approach that uses the
• 13:01segmentations for both the MRI
• 13:02and the ultrasound.
• 13:04Here we incorporated a geometric
• 13:06model,
• 13:07to model the deformation of
• 13:08the gland during the biopsy
• 13:10procedure to improve this image
• 13:12fusion process.
• 13:14Once we have the needle
• 13:16biopsy specimens, a challenge is
• 13:17then to determine if the
• 13:18full extent of tissue in
• 13:20that gland
• 13:21when this biopsy constitutes only
• 13:23a small fraction of the
• 13:24gland sampling.
• 13:26So using AI, we can
• 13:27then map this histopathology
• 13:29results back to the MRI
• 13:31to fill in the gaps.
• 13:33In this way, by identifying
• 13:34tissue with similar properties to
• 13:35the needle biopsy specimens, we
• 13:37can provide a full three
• 13:38d comprehensive assessment of disease
• 13:40risk throughout the gland.
• 13:43Also, as Tiba already mentioned,
• 13:46segmentation is great, but the
• 13:48real,
• 13:49critical practical question is, can
• 13:51we trust our predictions? So
• 13:52we've been at the forefront
• 13:53of developing methods to try
• 13:55to quantify uncertainty. And as
• 13:56you can see in the
• 13:57picture here, our frequency domain
• 13:59dropout here performs better in
• 14:01identifying the segmentation errors.
• 14:03So in pathology,
• 14:05one of the things that
• 14:06we're interested in is segmenting
• 14:07tissue and whole slide imaging.
• 14:08One of the major limitations
• 14:10of AI is that approaches
• 14:11are sensitive to how the
• 14:12data is oriented.
• 14:14So So as as that
• 14:15data rotates, biomarkers actually rotate
• 14:17around and they change position,
• 14:19which is bad for our
• 14:19performance.
• 14:21We've developed this new method
• 14:22that is able to robustly
• 14:24handle these features. So as
• 14:25the image rotates, the biomarker
• 14:27stays stable.
• 14:28Once we have these stable
• 14:29biomarkers we can then robustly
• 14:31segment the pathology images in
• 14:33an unsupervised manner.
• 14:36So finally, we're seeking to
• 14:37apply these image analysis approaches
• 14:39to treatment imaging such as
• 14:41theranostics
• 14:42And so these show some
• 14:43initial results segmenting tumor METs
• 14:46in,
• 14:47PSMA PET, and we're trying
• 14:49to quantify tumor burden throughout
• 14:50the body making this a
• 14:52really time consuming task more
• 14:54efficient and precise.
• 14:56So in summary, image analysis
• 14:58integrates multimodal data
• 15:00powered by AI is really
• 15:01a critical component in our
• 15:02toolbox for fight, fighting prostate
• 15:04cancer.
• 15:05So thank you for your
• 15:06time, and as we'll see
• 15:07next with Nisha that these
• 15:09similar methods can be applied
• 15:10also to breast cancer. So
• 15:12thank you.
• 15:18Hi, everyone. Again, my name
• 15:20is Nisha Dvornick, and thanks,
• 15:22first to the Imaging Institute
• 15:24leaders for this opportunity to
• 15:25share some of our work
• 15:26today in breast cancer imaging.
• 15:28So breast cancer is one
• 15:30of the most common and
• 15:30deadly cancers worldwide
• 15:32and the lifetime risk for
• 15:33women here in the US
• 15:34is about one in eight.
• 15:37Breast imaging plays a critical
• 15:38role from screening, to treatment
• 15:40of breast cancer.
• 15:42For example, mammography,
• 15:43type of X-ray imaging is
• 15:45the most commonly used modality
• 15:46for breast cancer screening and
• 15:47also used for diagnostic workup.
• 15:50MRI is used for high
• 15:52risk screening. We all heard
• 15:53with Todd Constable's work that
• 15:55eventually this affordable MRI will
• 15:56be for all women for
• 15:58screening, but for now it's
• 15:59just for high risk screening,
• 16:01for diagnosis, and also treatment
• 16:02planning and monitoring to see
• 16:04the extent of tumors.
• 16:06And finally, ultrasound imaging is
• 16:07used for for screening in
• 16:08some circumstances, diagnosis, and for
• 16:11biopsy guidance.
• 16:13Now analyzing all of these
• 16:15images faces a number of
• 16:16challenges including the large quantity
• 16:18of data, a growing shortage
• 16:19of breast radiologists, and subjective
• 16:21interpretation that depends on expertise.
• 16:25So to address these challenges,
• 16:27our group is developing novel
• 16:28approaches to enhance the analysis
• 16:30of breast images
• 16:31and we're adopting modern AI
• 16:33methods such as contrastive pre
• 16:35training to learn how to
• 16:36best model our data,
• 16:39utilizing multimodal image and text
• 16:41information,
• 16:42and leveraging domain knowledge of
• 16:43the imaging process to better
• 16:45constrain the image analysis models.
• 16:48So I'd like to just
• 16:49introduce a bit this idea
• 16:50of contrastive pre training for
• 16:52learning how to model or
• 16:53best represent this these imaging
• 16:55data.
• 16:56So the goal here is
• 16:57to learn a mapping
• 16:59oh, this is gone a
• 17:00mapping which is this image
• 17:01encoder,
• 17:02from the image to an
• 17:03embedding space where the image
• 17:05data will then be organized
• 17:06in a way that's easier
• 17:07to analyze in downstream tasks.
• 17:09So say we have this
• 17:10mammogram, it gets input through
• 17:11our image encoder and it
• 17:12maps to some embedding,
• 17:14and now we have another
• 17:16image, for the same patient
• 17:17but a different view, goes
• 17:19through this encoder, gets a
• 17:20different embedding,
• 17:21and now we have a
• 17:22third mammogram that comes from
• 17:24a different patient and goes
• 17:25through the same encoder and
• 17:26lands right there at the
• 17:27green triangle.
• 17:28So now the idea in
• 17:30contrastive learning is to learn
• 17:31this embedding where similar data
• 17:32are going to live closer
• 17:33together in this high dimensional
• 17:35space and dissimilar data are
• 17:36going to live farther apart.
• 17:38So the contrastive learning algorithm
• 17:40in this case is gonna
• 17:41work to align these blue
• 17:42circles that represent the images
• 17:43from the same patient
• 17:45while seeking to push apart
• 17:46data that come from different
• 17:47patients.
• 17:48And once we've learned this
• 17:49embedding and this useful representation
• 17:51of the data, we can
• 17:52build more accurate models for
• 17:54different image analysis tasks on
• 17:56top of it.
• 17:57So we use this image
• 17:59contrast to pre training approach
• 18:00to improve the identification of
• 18:02abnormal digital breast tumor synthesis
• 18:04scans, also known as three
• 18:05d mammography.
• 18:06And here our framework looks
• 18:07to align an image slice
• 18:08from a tonal volume with
• 18:10other slices from the same
• 18:11patient while pushing away the
• 18:12images from the other patients.
• 18:14After we've performed this contrastive
• 18:16pre training, we fine tune
• 18:18the model to learn to
• 18:18predict whether an input tomosynthesis
• 18:20slice is normal or abnormal.
• 18:23And as we can see
• 18:24in this performance,
• 18:25ROC curves here
• 18:27compared to the other pre
• 18:28training approaches,
• 18:30down below our model in
• 18:31red performs the best
• 18:33and in particular we achieved
• 18:34a very high negative predictive
• 18:35value suggesting we might be
• 18:37able to use this to
• 18:38filter out negative cases and
• 18:39reduce the radiologists workload.
• 18:43We then extended this contrastive
• 18:44pre training approach to align
• 18:46images and radiology reports. So
• 18:48this general method of aligning
• 18:50image and text data is
• 18:51known as contrastive language pre
• 18:52training or CLIP.
• 18:54So now in addition to
• 18:55the image encoder, we now
• 18:56have this text encoder to
• 18:57map the radiology reports into
• 18:59now this shared embedding space.
• 19:01And the contrastive,
• 19:03learning algorithm is going to
• 19:04now try to
• 19:05align the paired image and
• 19:07radiology reports and at the
• 19:09same time push away other
• 19:10reports that were not related
• 19:11to the target image
• 19:13and push away other images
• 19:14that are not related to
• 19:15the target report.
• 19:17So we use this CLIP
• 19:18framework to perform a multi
• 19:19view and multi scale alignment
• 19:21from mammography data
• 19:23where our model considers again
• 19:24this multi view image alignment
• 19:26at top, similar to the
• 19:27tone synthesis model. And then
• 19:29we add in this global
• 19:31image report alignment using the
• 19:32CLIP framework and also local
• 19:34level image report alignment
• 19:36where we learn to match
• 19:37sentence level information in the
• 19:38radiology report with the most
• 19:40relevant image patch in the
• 19:41mammogram and vice versa.
• 19:44So after we perform this
• 19:45pretraining,
• 19:46we can then fine tune
• 19:47the model to do different
• 19:48image analysis tasks including,
• 19:51prediction of the BI RADS
• 19:52category,
• 19:53breast density, and cancer. And
• 19:56we can see our approach
• 19:57in the blue gray on
• 19:58the very right of each
• 19:59of these plots, performs the
• 20:00best for each of these
• 20:01model learning scenarios.
• 20:04Finally, we leverage the geometry
• 20:06of the multi view imaging
• 20:08process to learn how to
• 20:09align local image features before
• 20:10we're just looking at kind
• 20:11of the global image. And
• 20:13we can see here, at
• 20:14the bottom that red dot,
• 20:15this, small region of interest
• 20:17actually corresponds to multiple locations
• 20:19in the blue tube on
• 20:20the left,
• 20:22that, goes through the same
• 20:23slice on the breast. And
• 20:25so we use this geometric
• 20:26constraint to perform this contrast
• 20:27of pretraining to align each
• 20:29patch in one view with
• 20:30each corresponding slice in the
• 20:31other view. And using this
• 20:33local alignment, we again see
• 20:34improvement in this BI RADS
• 20:36category prediction.
• 20:38So finally, I'm just gonna
• 20:40touch on our breast MRI
• 20:41work, where we're looking to
• 20:42improve the segmentation of tumors
• 20:43with additional knowledge of the
• 20:44imaging process. So here's kind
• 20:46of a standard AI model
• 20:48for segmentation where the image
• 20:49goes into a unit and
• 20:50then we get our prediction
• 20:51for our tumor mask.
• 20:53But in our approach, we're
• 20:54going to incorporate the timing
• 20:55of these contrast enhanced MRI
• 20:57to modulate the model parameters
• 20:59and we saw a nice
• 21:00increase in our early results
• 21:02in the segmentation performance.
• 21:04So to conclude, I hope
• 21:05I've given you all an
• 21:06idea of how modern AI
• 21:07approaches can help, enhance breast
• 21:09image analysis. And if you
• 21:10have any ideas on how
• 21:12this might apply to your
• 21:12data or some specific task,
• 21:14I'd be very happy to
• 21:15discuss further. Thank you.