image analysis research

Your browser does not support javascript. Some site functionality may not work as expected.

• Images from UW Libraries
• Open Images
• Image Analysis
• Citing Images
• University of Washington Libraries
• Library Guides
• Images Research Guide

Images Research Guide: Image Analysis

Analyze images.

Content analysis    

• What do you see?
• What is the image about?
• Are there people in the image? What are they doing? How are they presented?
• Can the image be looked at different ways?
• How effective is the image as a visual message?

Visual analysis  

• How is the image composed? What is in the background, and what is in the foreground?
• What are the most important visual elements in the image? How can you tell?
• How is color used?
• What meanings are conveyed by design choices?

Contextual information  

• What information accompanies the image?
• Does the text change how you see the image? How?
• Is the textual information intended to be factual and inform, or is it intended to influence what and how you see?
• What kind of context does the information provide? Does it answer the questions Where, How, Why, and For whom was the image made?

Image source  

• Where did you find the image?
• What information does the source provide about the origins of the image?
• Is the source reliable and trustworthy?
• Was the image found in an image database, or was it being used in another context to convey meaning?

Technical quality  

• Is the image large enough to suit your purposes?
• Are the color, light, and balance true?
• Is the image a quality digital image, without pixelation or distortion?
• Is the image in a file format you can use?
• Are there copyright or other use restrictions you need to consider? 

  developed by Denise Hattwig , [email protected]

More Resources

National Archives document analysis worksheets :

• Photographs
• All worksheets

Visual literacy resources :

• Visual Literacy for Libraries: A Practical, Standards-Based Guide   (book, 2016) by Brown, Bussert, Hattwig, Medaille ( UW Libraries availability )
• 7 Things You Should Know About... Visual Literacy ( Educause , 2015 )
• Keeping Up With... Visual Literacy  (ACRL, 2013)
• Visual Literacy Competency Standards for Higher Education (ACRL, 2011)
• Visual Literacy White Paper  (Adobe, 2003)
• Reading Images: an Introduction to Visual Literacy (UNC School of Education)
• Visual Literacy Activities (Oakland Museum of California)
• << Previous: Open Images
• Next: Citing Images >>
• Last Updated: Aug 6, 2024 12:41 PM
• URL: https://guides.lib.uw.edu/newimages

Quick Links:

Introduction to Image Analysis: Understanding the Basics and Applications

31 Jul 2023

Sandeep Kulkarni

Founder & CEO

Introduction:

Following are some of the basic ideas in image analysis:, image acquisition:, pre-processing:, segmentation:, feature extraction:, object recognition:, image classification:.

Read Also - The Basic Of Image Processing

Image Analysis Applications:

Security and surveillance:, robotics and automation:, agriculture:, pharmaceutical industry:, quality control and inspection:, biometrics:, how has image analysis reshaped the pharmaceutical industry.

Read Also - Image Filtering Techniques in Image Processing

How Has Image Analysis Advanced Scientific Research?

Cellular and molecular biology:, medicine and biomedical research:, neuroscience:, ecology and environmental sciences:, astronomy and astrophysics:, materials science:, social sciences:, how image analysis has made social media platforms more user-friendly, different image analysis software used:, scikit-image:, tensorflow:, cellprofiler:, deeplabcut:.

Read Also - Particle Size Analysis: Importance & Applications 

Future of Image Analysis:

Sandeep Kulkarni, Founder & CEO

Sandeep Kulkarni is the founder & CEO of ImageProVision Technology. With over 3 decades of experience behind him, he is your 'go-to' man in the image analysis sector.

Articles, you may also like.

Understanding Image Filtering Techniques in Image Processing

How AI/ML is Used in Particle Analysis Characterization

Beyond Enumeration: ipvMicrobe Colony Counters

Submit your details to schedule your demo..

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Thank you for reaching out. We will get back to you soon.

Study at Cambridge

About the university, research at cambridge.

• Events and open days
• Fees and finance
• Student blogs and videos
• Why Cambridge
• Qualifications directory
• How to apply
• Fees and funding
• Frequently asked questions
• International students
• Continuing education
• Executive and professional education
• Courses in education
• How the University and Colleges work
• Visiting the University
• Term dates and calendars
• Video and audio
• Find an expert
• Publications
• International Cambridge
• Public engagement
• Giving to Cambridge
• For current students
• For business
• Colleges & departments
• Libraries & facilities
• Museums & collections
• Email & phone search

Cambridge Image Analysis

• Arts Restoration
• Inverse Problems
• Biomedical Imaging and Analysis
• Machine Learning
• Optimisation
• Registration and Motion Estimation
• Remote Sensing
• Segmentation
• COVID-19 Collaboration
• Historic Collaboration
• Current Term
• Introduction to Nonlinear Spectral Analysis (Lent 2021)
• Inverse Problems (Michaelmas 2020)
• Inverse Problems (Michaelmas 2019)
• Bayesian Inverse Problems (Lent 2019)
• Introduction to Optimal Transport (Lent 2019)
• Inverse Problems (Lent 2018)
• Inverse Problems in Imaging (Michaelmas 2018)
• Inverse Problems in Imaging (2016)

"I’m pleased to welcome you to the Cambridge Image Analysis Group (CIA). Here, we blend mathematics and innovation to solve impactful problems in imaging, machine learning and in a diverse range of applications. Join us on this exciting journey of discovery. Welcome to the CIA!"                                                                                               — Professor Carola-Bibiane Schönlieb

Welcome to the Cambridge Image Analysis (CIA) Group in DAMTP . Our group specialises in theory and methodology development to solve intricate problems, ranging from digital image and video processing to inverse problems and partial differential equations, optimisation algorithms, mathematical modelling, and machine learning. Join us in using the power of mathematics to solve challenging problems and create innovative solutions for the future.

Some topic we work on:

• Variational methods and partial differential equations
• Image restoration
• Image segmentation and object tracking
•   Image registration and motion compensation
• Semi-supervised learning, reinforcement learning etc.
• Graph and hypergraph learning
• Computer vision including low-level vision
• Optimisation (e.g., learning to optimise techniques)
• Generative AI (diffusion models, GANs, normalising Flows)
• Biomedical imaging (e.g.,MRI,  PET/SPECT, microscopy imaging)
• Computational photography
• Full blood counts
• Arts restoration
• Remote sensing
• Traffic flow analysis
• Large scale computing
• Multi-modal analysis

For more information on current research projects, please visit our research page and the project websites for the Cambridge Mathematics of Information in Healthcare Hub (CMIH) and the Cantab Capital Institute for the Mathematics of Information (CCIMI) , Blood Counts and Mathematics of Deep Learning (Maths4DL).

Tweets by CamImaging

© 2024 University of Cambridge

• University A-Z
• Contact the University
• Accessibility
• Freedom of information
• Terms and conditions
• Undergraduate
• Spotlight on...
• About research at Cambridge

Pattern Recognition and Image Analysis

• Encompasses various topics, including the identification of patterns or regularities in data and computer vision with a focus on processing and interpreting visual information contained in images.
• Embraces the problems of analyzing and evaluating poorly formalized, poorly structured, incomplete, contradictory, and noisy information
• One of the top ten global periodicals on image analysis and pattern recognition.
• Welcomes manuscripts from all countries.
• Igor A. Sokolov

Societies and partnerships

Latest issue

Volume 34, Issue 2

Latest articles

Detection system of landscape’s unnatural changes by satellite images based on local areas.

• Alexander Mixailovich Nedzved

Automatic Analysis of Walking Steps

• Olga Nedzvedz
• Alexander Nedzved

A Writer-Dependent Approach to Offline Signature Verification Based on One-Class Support Vector Machine

• V. V. Starovoitov
• U. Yu. Akhundjanov

Diabetic Retinopathy Fundus Image Classification Using Ensemble Methods

• Marina M. Lukashevich

Construction of a Semiautomatic Contour of Areal Objects on Hyperspectral Satellite Images

• Alexei Belotserkovsky

Journal information

• ACM Digital Library
• EI Compendex
• Emerging Sources Citation Index
• Google Scholar
• Japanese Science and Technology Agency (JST)
• OCLC WorldCat Discovery Service
• Russian Science Citation Index
• TD Net Discovery Service
• UGC-CARE List (India)

Rights and permissions

Editorial policies

© Pleiades Publishing, Ltd.

• Find a journal
• Publish with us
• Track your research

AI for Image Analysis

How do we use AI to extract useful information from images?

Our world is highly visual. We derive most of our information about the world through our eyes.

The spaces that we navigate, the faces we interact with, and the documents we read are all processed visually. As a result, digital images are one of the most valuable types of unstructured data for modern AI.

In the last two articles of this multi-part series on The AI Developer’s Toolkit , I introduced you to the top AI tools for audio analysis and audio synthesis . In this article, I’ll introduce you to the three most popular AI tools for image analysis .

Image Classification

Image classification allows us to assign an image to two or more labeled categories. It answers the question, “what is contained in this image?”

For example, we can use image classification to tag the content contained in an image. We provide the image-classification model with an image as input. Then the model produces a category label and a confidence score as output.

Image classification is useful anytime you are trying to assign a categorical label (or multiple tags) to a collection of images. For example:

• auto-tagging images on social-media posts
• detecting product defects via visual inspection
• diagnosing medical issues like detecting certain types of skin cancer

Object Detection

Object detection allows us to identify the location of various objects in an image. It answers the question, “where are the objects located in this image?”

For example, we can use object detection to identify various items contained in an image. We provide the object-detection model with an image as input. Then the model produces the coordinates of a bounding box for each object in the image as output.

Object detection is useful anytime you have images with multiple objects that need to be located. For example:

• counting the number objects in an photo
• detecting people in surveillance videos
• detecting various obstacles in a self-driving car

Face Recognition

Face recognition allows us to identify a person contained in an image by their facial features. It answers the question, “who is in this image?”

For example, we can use face recognition to determine who is contained in our photos. We provide the face-recognition model with an image as input. Then the model produces the identity of the person contained in the image as output.

Face recognition is useful anytime you need to identify people in images. For example:

• identifying customers as they enter your store
• recognizing the occupants of your office building
• tagging your friends in photos on social-media

Other Tools

Beyond the three examples that we’ve seen so far, there are also a variety of other image-analysis tasks. For example:

• Reverse image Search – which allows us to find images that are visually similar to a source image
• Image captioning – which generates a text description of what is contained in an image
• Image segmentation – which is like object detection but assigns a type of object to every pixel in an image
• Face-analysis tools – which allow us to detect faces, compare faces, detect facial landmarks, determine gender and age, detect facial features, and classify emotions
• Body-analysis tools – which allow us to estimate pose, recognize gestures, count fingers, and detect adult or racy content
• Document-analysis tools which allow us to extract printed text, handwritten text, form data, and tables from documents

As we can see, image-analysis tools allow us to extract useful information from digital images.

If you’d like to learn how to use all of the tools listed above, please watch my online course: The AI Developer’s Toolkit .

The future belongs who those who invest in AI today. Don’t get left behind!

• Cookies Policy

© 2024 Matthew Renze

Image Analysis

Most simply put, image analysis is the extraction of meaningful information from images. The images come from many sources and are examined in many ways. Computers are especially useful in image analysis.

Research Area Faculty

The faculty researchers in this area exemplify the collaborative nature of the work done at Cornell Engineering.

C Thomas Avedisian

Jonathan T. Butcher

Edwin (Todd) Cowen

Peter Doerschuk

David Lee Hysell

Stephen Robert Marschner

Alyosha Christopher Molnar

David Anthony Muller

Sriramya Duddukuri Nair

Anthony P. Reeves

Mert Sabuncu

Dmitry Savransky

Chris Schaffer

Warren R. Zipfel

Research groups.

• Cornell-Cantabria Exchange Program
• David Muller Group
• Jicamarca Radio Observatory
• Kavli Institute at Cornell for Nanoscale Science
• Molnar Group
• Nair's Research Group
• Publications of C. Thomas Avedisian
• Research Gate Archive of Cowen's Publications
• Stephen Marschner Homepage
• Upper Atmospheric Research
• Vision and Image Analysis Group (VIA)

Explore More Research Areas

Biotechnology

Biomedical Engineering

Molecular Biotechnology

Image Analysis

We develop techniques to help turn medical images into medical insights that can be used for downstream prediction of health outcomes. Specifically, we seek to build segmentation tools to automatically detect potential biomarkers of disease activity for varying anatomies and volumetric imaging techniques.

Musculoskeletal MRI

We have developed, validated, and open-sourced algorithms to perform accurate segmentation of the articular cartilage and meniscus from knee MRI scans. Such segmentations are used to quantify morphological biomarkers as well as to establish quantitative MRI values from MRI sequences that we have previously developed. To mitigate the paucity of labeled training datasets for image segmentation, we are developing data-efficient techniques using principles of self-supervision. In conjunction with the datasets that we have shared with the research community that enable accelerated MRI, we are developing end-to-end techniques for facilitating rapid MRI acquisition and automated analysis.

Select Publications:

• Desai A, Caliva F, Iriondo C, Khosravan N, Mortazi A, Jambawalikar S, Torigian D, Ellerman J, Akcakaya M, Bagci U, Tibrewala R, Flament I, O’Brian M, Majumdar S, Perslev M, Pai A, Igel C, Dam E, Gaj S, Yang M, Nakamura K, Li X, Deniz C, Juras V, Regatte, Gold G, Hargreaves B, Pedoia V, and Chaudhari A. The International Workshop on Osteoarthritis Imaging Knee MRI Segmentation Challenge: A Multi-Institute Evaluation and Analysis Framework on a Standardized Dataset. Radiology: Artificial Intelligence (2021) 3:3. doi: 10.1148/ryai.2021200078
• Wirth W, Eckstein F, Kemnitz J, Baumgartner C, Konukoglu E, Furst D, and Chaudhari A. Accuracy and Longitudinal Reproducibility of Quantitative Femorotibial Cartilage Measures Derived from Automated U-Net-based Segmentation of Two Different MRI Contrasts – Data from the Osteoarthritis Initiative Healthy Reference Cohort. Magnetic Resonance Materials in Physics, Biology and Medicine (2021). 34(3):337-354. doi: 10.1007/s10334-020-00889-7
• Eckstein F, Chaudhari A, Fuerst D, Gaisberger M, Kemnitz J, Baumgartner C, Konukoglu E, Hunter D, Wirth W. A Deep Learning Automated Segmentation Algorithm Accurately Detects Differences in Longitudinal Cartilage Thickness Loss – Data from the FNIH Biomarkers Study of the Osteoarthritis Initiative. Arthritis Care & Research (2020). doi: 10.1002/acr.24539
• Schmidt A, Desai A, Watkins L, Crowder H, Mazzoli V, Rubin E, Lu Q, Black M, Kogan F, Gold G, Hargreaves B, and Chaudhari A. Generalizability of Deep-Learning Segmentation Algorithms for Measuring Cartilage and Meniscus Morphology and T2 Relaxation Times. Intl Soc Magn Reson Med, (virtual), 2021
• Dominic F, Desai A, Schmidt A, Rubin A, Gold G, Hargreaves B, and Chaudhari A. Self-Supervised Deep Learning for Knee MRI Segmentation using Limited Labeled Training Datasets. Intl Soc Magn Reson Med, (virtual), 2021.
• Desai A, Barbieri M, Mazzoli V, Rubin E, Black M, Watkins E, Gold G, Hargreaves B, and Chaudhari A. DOSMA: A Deep learning, Open-Source Framework for Musculoskeletal MRI Analysis. Intl Soc Magn Reson Med, Montreal, 2019.

Body Composition with CT

Using routine abdominal computed tomography (CT) scans, we extract quantitative biomarkers that depict the status of muscle and adipose tissue. Such biomarkers have been linked to future disease onset, post-surgical outcomes, and all-cause mortality.

• Desai A, Boutin R, Tan J, Lenchik L, and Chaudhari A. An Evaluation of Automated Body Composition Analysis from Abdominal Computed Tomography Scans using Deep Learning. Society of Advanced Body Imaging Annual Meeting, (virtual), 2020.
• Boutin RD, Barnard RT, Kim J, Tan JC, Chaudhari A, Lenchik L. Opportunistic CT Assessment of Biological Aging: Comparing 2D vs. 3D Metrics for Muscle and Adipose Tissue. Radiological Society of North America, Chicago, 2021
• Zambrano JM, Chaudhari A, Wentland A, Jeffrey B, Rubin D, and Patel B. Opportunistic Screening for Ischemic Heart Disease Risk Using Abdominopelvic Computed Tomography and Medical Record Data: A Multimodal Explainable Artificial Intelligence Approach. Society of Abdominal Radiology 2021 (virtual).

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Perspective
• Published: 28 June 2012

Fiji: an open-source platform for biological-image analysis

• Johannes Schindelin 1   nAff10 ,
• Ignacio Arganda-Carreras 2 ,
• Erwin Frise 3 ,
• Verena Kaynig 4 ,
• Mark Longair 5 ,
• Tobias Pietzsch 1 ,
• Stephan Preibisch 1   nAff10 ,
• Curtis Rueden 6 ,
• Stephan Saalfeld 1 ,
• Benjamin Schmid 7   nAff10 ,
• Jean-Yves Tinevez 8 ,
• Daniel James White 1 ,
• Volker Hartenstein 9 ,
• Kevin Eliceiri 6 ,
• Pavel Tomancak 1 &
• Albert Cardona 5   nAff10  

Nature Methods volume  9 ,  pages 676–682 ( 2012 ) Cite this article

141k Accesses

33k Citations

150 Altmetric

Metrics details

Fiji is a distribution of the popular open-source software ImageJ focused on biological-image analysis. Fiji uses modern software engineering practices to combine powerful software libraries with a broad range of scripting languages to enable rapid prototyping of image-processing algorithms. Fiji facilitates the transformation of new algorithms into ImageJ plugins that can be shared with end users through an integrated update system. We propose Fiji as a platform for productive collaboration between computer science and biology research communities.

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 12 print issues and online access

251,40 € per year

only 20,95 € per issue

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

ilastik: interactive machine learning for (bio)image analysis

High-throughput image processing software for the study of nuclear architecture and gene expression

Sopa: a technology-invariant pipeline for analyses of image-based spatial omics

Turing, A.M. The chemical basis of morphogenesis. 1953. Bull. Math. Biol. 52 , 153–197, discussion 119–152 (1990).

Article   CAS   Google Scholar  

Altschul, S.F., Gish, W., Miller, W., Myers, E.W. & Lipman, D.J. Basic local alignment search tool. J. Mol. Biol. 215 , 403–410 (1990).

Myers, E.W. et al. A whole-genome assembly of Drosophila . Science 287 , 2196–2204 (2000).

Neumann, B. et al. Phenotypic profiling of the human genome by time-lapse microscopy reveals cell division genes. Nature 464 , 721–727 (2010).

Collinet, C. et al. Systems survey of endocytosis by multiparametric image analysis. Nature 464 , 243–249 (2010).

Shariff, A., Kangas, J., Coelho, L.P., Quinn, S. & Murphy, R.F. Automated image analysis for high-content screening and analysis. J. Biomol. Screen. 15 , 726–734 (2010).

Article   Google Scholar  

Megason, S.G. & Fraser, S.E. Imaging in systems biology. Cell 130 , 784–795 (2007).

Keller, P.J., Schmidt, A.D., Wittbrodt, J. & Stelzer, E.H. Reconstruction of zebrafish early embryonic development by scanned light sheet microscopy. Science 322 , 1065–1069 (2008).

Fowlkes, C.C. et al. A quantitative spatiotemporal atlas of gene expression in the Drosophila blastoderm. Cell 133 , 364–374 (2008).

Anderson, J.R. et al. A computational framework for ultrastructural mapping of neural circuitry. PLoS Biol. 7 , e1000074 (2009).

Saalfeld, S., Cardona, A., Hartenstein, V. & Tomancak, P. As-rigid-as-possible mosaicking and serial section registration of large ssTEM datasets. Bioinformatics 26 , i57–i63 (2010).

Murray, J.I. et al. Automated analysis of embryonic gene expression with cellular resolution in C. elegans . Nat. Methods 5 , 703–709 (2008).

Fernandez, R. et al. Imaging plant growth in 4D: robust tissue reconstruction and lineaging at cell resolution. Nat. Methods 7 , 547–553 (2010).

Bao, Z. et al. Automated cell lineage tracing in Caenorhabditis elegans . Proc. Natl. Acad. Sci. USA 103 , 2707–2712 (2006).

Long, F., Peng, H., Liu, X., Kim, S.K. & Myers, E. A 3D digital atlas of C. elegans and its application to single-cell analyses. Nat. Methods 6 , 667–672 (2009).

Peng, H. et al. BrainAligner: 3D registration atlases of Drosophila brains. Nat. Methods 8 , 493–500 (2011).

Swedlow, J.R. & Eliceiri, K.W. Open source bioimage informatics for cell biology. Trends Cell Biol. 19 , 656–660 (2009).

Peng, H. Bioimage informatics: a new area of engineering biology. Bioinformatics 24 , 1827–1836 (2008).

Abramoff, M.D., Magalhaes, P.J. & Ram, S.J. Image processing with ImageJ. Biophotonics International 11 , 36–42 (2004).

Google Scholar  

Carpenter, A.E. et al. CellProfiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biol. 7 , R100 (2006).

Peng, H., Ruan, Z., Long, F., Simpson, J.H. & Myers, E.W. V3D enables real-time 3D visualization and quantitative analysis of large-scale biological image data sets. Nat. Biotechnol. 28 , 348–353 (2010).

de Chaumont, F., Dallongeville, S. & Olivo-Marin, J.-C. ICY: a new open-source community image processing software in. IEEE Int. Symp. on Biomedical Imaging 234–237 (2011).

Berthold, M.R. et al. KNIME: the Konstanz Information Miner. in Studies in Classification, Data Analysis, and Knowledge Organization (GfKL 2007) 319–326 (Springer, 2007).

Cardona, A. et al. TrakEM2 software for neural circuit reconstruction. PLoS One (in the press).

Schmid, B., Schindelin, J., Cardona, A., Longair, M. & Heisenberg, M. A high-level 3D visualization API for Java and ImageJ. BMC Bioinformatics 11 , 274 (2010).

Preibisch, S., Saalfeld, S., Schindelin, J. & Tomancak, P. Software for bead-based registration of selective plane illumination microscopy data. Nat. Methods 7 , 418–419 (2010).

Preibisch, S., Tomancak, P. & Saalfeld, S. in Proc. ImageJ User and Developer Conf. 1 , 72–76 (2010).

Matas, J., Chum, O., Urban, M. & Pajdla, T. Robust wide baseline stereo from maximally stable extremal regions. Image Vis. Comput. 22 , 761–767 (2004).

Ibanez, L., Schroeder, W., Ng, L. & Cates, J. The ITK Software Guide (Kitware Inc., 2005).

Köthe, U. Reusable software in computer vision. in Handbook of Computer Vision and Applications Vol. 3 (eds. Jähne, B., Haussecker, H. & Geissler P.) 103–132 (San Diego: Academic Press, 1999).

Preibisch, S., Saalfeld, S. & Tomancak, P. Globally optimal stitching of tiled 3D microscopic image acquisitions. Bioinformatics 25 , 1463–1465 (2009).

Cardona, A. et al. An integrated micro- and macroarchitectural analysis of the Drosophila brain by computer-assisted serial section electron microscopy. PLoS Biol. 8 , e1000502 (2010).

Bock, D.D. et al. Network anatomy and in vivo physiology of visual cortical neurons. Nature 471 , 177–182 (2011).

Kaynig, V., Fischer, B., Müller, E. & Buhmann, J.M. Fully automatic stitching and distortion correction of transmission electron microscope images. J. Struct. Biol. 171 , 163–173 (2010).

Saalfeld, S., Fetter, R., Cardona, A. & Tomancak, P. Elastic volume reconstruction from series of ultrathin microscopy sections. Nature Methods advance online publication, doi:10.1038/nmeth.2072 (10 June 2012).

Arganda-Carreras, I. et al. Consistent and elastic registration of histological sections using vector-spline regularization. in Lecture Notes in Computer Science 4241 , 85–95 (Springer, 2006).

Lowe, D.G. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60 , 91–110 (2004).

Sethian, J.A. Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Science (Cambridge University Press, 1999).

Kass, M., Witkin, A. & Terzopoulos, D. Snakes: active contour models. Int. J. Comput. Vis. 1 , 321–331 (1988).

Hall, M. et al. The WEKA data mining software: an update. SIGKDD Explor. 11 , 10–18 (2009).

Kaynig, V., Fuchs, T.J. & Buhmann, J.M. Geometrical consistent 3D tracing of neuronal processes in ssTEM data. Med. Image. Comput. Comput. Assist. Interv. 13 , 209–216 (2010).

PubMed   Google Scholar  

Cardona, A. et al. Identifying neuronal lineages of Drosophila by sequence analysis of axon tracts. J. Neurosci. 30 , 7538–7553 (2010).

Longair, M.H., Baker, D.A. & Armstrong, J.D. Simple Neurite Tracer: open source software for reconstruction, visualization and analysis of neuronal processes. Bioinformatics 27 , 2453–2454 (2011).

Edelstein, A., Amodaj, N., Hoover, K., Vale, R. & Stuurman, N. Computer control of microscopes using μManager. in Current Protocols in Molecular Biology (John Wiley & Sons, Inc., 2010).

Download references

Acknowledgements

We thank W. Rasband for developing ImageJ and helping thousands of scientists, those who contributed to the Fiji movement by financing and organizing the hackathons, namely G.M. Rubin for hackathons at Janelia Farm, I. Baines for hackathons at Max Planck Institute of Molecular Cell Biology and Genetics in Dresden, R. Douglas for a hackathon at Institute of Neuroinformatics in Zurich, F. Peri and K. Miura for the hackathon at European Molecular Biology Laboratory, and International Neuroinformatics Coordinating Facility for Fiji image-processing school, W. Pereanu for the confocal image of the larval fly brain, M. Sarov for the confocal scan of C. elegans larva, the scientists who released their code under open-source licenses and made the Fiji project possible. We want to thank Carl Zeiss Microimaging for access to the SPIM demonstrator. K.E. and C.R. were supported by US National Institutes of Health grant RC2GM092519. J.S. and P.T. were funded by Human Frontier Science Program Young Investigator grant RGY0083. P.T. was supported by The European Research Council Community′s Seventh Framework Programme (FP7/2007-2013) grant agreement 260746.

Author information

Johannes Schindelin, Stephan Preibisch, Benjamin Schmid & Albert Cardona

Present address: Present addresses: Laboratory for Optical and Computational Instrumentation, University of Wisconsin at Madison, Madison, Wisconsin, USA (J.S.), Max Planck Institute of Molecular Cell Biology and Genetics, Dresden, Germany (B.S.) and Janelia Farm Research Campus, Howard Hughes Medical Institute, Ashburn, Virginia, USA (S.P. and A.C.).,

Authors and Affiliations

Max Planck Institute of Molecular Cell Biology and Genetics, Dresden, Germany

Johannes Schindelin, Tobias Pietzsch, Stephan Preibisch, Stephan Saalfeld, Daniel James White & Pavel Tomancak

Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA

Ignacio Arganda-Carreras

Department of Genome Dynamics, Berkeley Drosophila Genome Project, Lawrence Berkeley National Laboratory, Berkeley, California, USA

Erwin Frise

Department of Computer Science of the Swiss Federal Institute of Technology Zurich, Zurich, Switzerland

Verena Kaynig

Institute of Neuroinformatics of the University of Zurich and Swiss Federal Institute of Technology Zurich, Zurich, Switzerland

Mark Longair & Albert Cardona

Laboratory for Optical and Computational Instrumentation, University of Wisconsin at Madison, Madison, Wisconsin, USA

Curtis Rueden & Kevin Eliceiri

Department of Neurobiology and Genetics, University of Wurzburg, Wurzburg, Germany

Benjamin Schmid

Institut Pasteur, Imagopole, La plate-forme d'imagerie dynamique, Paris, France

Jean-Yves Tinevez

Department of Molecular, Cell and Developmental Biology, University of California, Los Angeles, California, USA

Volker Hartenstein

You can also search for this author in PubMed   Google Scholar

Corresponding authors

Correspondence to Pavel Tomancak or Albert Cardona .

Ethics declarations

Competing interests.

The authors declare no competing financial interests.

Supplementary information

Supplementary text and figures.

Supplementary Figures 1–3 and Supplementary Table 1 (PDF 1230 kb)

Supplementary Video 1

Visualization of Fiji development. The video, produced using 'gource' tool in Git, visualizes the changes to Fiji source code repository from 15 March 2009 to 16 May 2009. The class hierarchy is visualized as a dynamic tree, the developers are flying pawns that extend rays to classes that they newly created or into which they introduced changes. Between 23 March and 3 April 2009 there was a Fiji hackathon in Dresden, Germany, marked by increased developer activity that carries over the period after the hackathon ended, the 'hackathon effect'. (MOV 10866 kb)

Supplementary Video 2

Visualization of SIFT-mediated stitching of large ssTEM mosaics. The ventral nerve cord of Drosophila first instar larva was sectioned and imaged in electron microscope as a series of overlapping image tiles. The video visualizes the process of reconstruction of such large section series on seven exemplary sections. The corresponding SIFT features that connect images within section and across section are shown as green dots, the residual error of their displacement at a given iteration of the global optimizer is shown as cyan line (iteration number and minimal, average and maximal error are shown in lower left corner). The global optimization proceeds section by section and at each step distributes the registration error equally across the increasing set of tiles. To emphasize the visualization effect all tiles within section are initially placed at the same location discarding their known configuration within section. (MOV 15759 kb)

Supplementary Video 3

Visualization of bead-based registration of multiview microscopy scan of Drosophila embryo. Drosophila embryo expressing His-YFP marker has been imaged in a spinning disc confocal microscope from 18 different angles improvising rotation using custom made sample chamber. The video visualizes the global optimization that is using local geometric bead descriptor matches to recover the shape of the embryo specimen. The bead descriptors (representing constellations of sub-resolution fluorescent beads added to the rigid agarose medium in which the embryo was mounted) are colored according to their displacement at each iteration of the optimizer (red, maximum displacement; green, minimum displacement). The nuclei of the embryo specimen are shown in grey. The displacement at each iteration averaged across all descriptors is shown in the lower left corner. (MOV 6422 kb)

Supplementary Video 4

Segmentation and tracking of nuclei in Drosophila embryo. Cellular blastoderm stage Drosophila embryo expressing His-YFP marker in all cells was imaged from five angles using SPIM throughout gastrulation. The video shows a result of segmentation and tracking algorithm that follows the movements of cells through the gastrulation process. The nuclei are colored according to the angle at which they were detected. (MOV 13736 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Schindelin, J., Arganda-Carreras, I., Frise, E. et al. Fiji: an open-source platform for biological-image analysis. Nat Methods 9 , 676–682 (2012). https://doi.org/10.1038/nmeth.2019

Download citation

Published : 28 June 2012

Issue Date : July 2012

DOI : https://doi.org/10.1038/nmeth.2019

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Transcriptionally active chromatin loops contain both ‘active’ and ‘inactive’ histone modifications that exhibit exclusivity at the level of nucleosome clusters.

• Stefan A. Koestler
• Madeleine L. Ball
• Robert White

Epigenetics & Chromatin (2024)

Chondrosarcoma evaluation using hematein-based x-ray staining and high-resolution 3D micro-CT: a feasibility study

• Alexandra S. Gersing
• Melanie A. Kimm
• Franz Pfeiffer

European Radiology Experimental (2024)

TrkB-mediated neuroprotection in female hippocampal neurons is autonomous, estrogen receptor alpha-dependent, and eliminated by testosterone: a proposed model for sex differences in neonatal hippocampal neuronal injury

• Vishal Chanana
• Pelin Cengiz

Biology of Sex Differences (2024)

Assessment of the FRET-based Teen sensor to monitor ERK activation changes preceding morphological defects in a RASopathy zebrafish model and phenotypic rescue by MEK inhibitor

• Giulia Fasano
• Stefania Petrini
• Antonella Lauri

Molecular Medicine (2024)

Assessing microbially mediated vivianite as a novel phosphorus and iron fertilizer

• Lordina Ekua Eshun
• Ana Maria García-López
• Antonio Delgado

Chemical and Biological Technologies in Agriculture (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Have questions about buying, selling or renting during COVID-19? Learn more

Market Reports

• Sellers lose their advantage, but lower rates may revive market competition (July 2024 Market Report)
• One-Third of Property Managers are Offering Concessions as Rental Market Cools

An apartment construction boom is giving renters more options and better deals

Zillow Home Value and Home Sales Forecast (June 2024)

Price Cuts Abound as Home Sellers See Inventory Piling Up (June 2024 Market Report)

The Rental Market Slowdown is Leveling Off (June 2024 Rental Market Report)

Market Moves Closer to Balance as Sellers Return and Buyers Balk (May 2024 Market Report)

Home value growth eases along with competition – price relief may be on the horizon.

Higher Wages, Steady Rental Market Have Allowed Renters to Catch Their Breath (May 2024 Rental Market Report)

The Housing Market Eased off the Accelerator in April as Mortgage Rates Spiked (April 2024 Market Report)

Home value growth and days on market slowed in April, while price cuts jumped

Renters Need to Make $80,000 to Comfortably Afford the Typical U.S. Rental (April 2024 Rental Market Report)

The Expensive Get More Expensive: Home Value Growth Tops in Highest-Price Markets (March 2024 Market Report)

Monthly appreciation spikes in expensive West Coast metros with meager options, stays cooler in areas where inventory has returned.

• July 2024 Housing Starts: Housing Starts Fall, Single Family Starts At Lowest Level Since April 2023
• Support Growing for Middle Housing
• Mortgage Rates Fell This Week On Exaggerated Recession Fears
• Mortgage Rates Fell This Week As Wage Inflation Moderates More Than Expected
• Luxury Home Values Are Rising Faster Than Typical Homes for the First Time in Years
• A $1 Million Starter Home is the Norm in 237 Cities
• Mortgage Rates Rebounded Slightly This Week Ahead of Key Inflation Report
• June 2024: New home Sales Fell Again Despite Easing Mortgage Rates

• Back to main menu
• BROWSE BY TOPIC BROWSE BY TOPIC
• Global IT Asset Management
• IT Security
• Cloud & Container Security
• Web App Security
• Certificate Security & SSL Labs
• Developer API
• Cloud Platform
• Start a discussion

Understanding the New Windows Secure Kernel Mode Elevation of Privilege Vulnerability (CVE-2024-21302)

Table of Contents

Exploitability, executive summary, detailed analysis, impact assessment, qualys qid coverage, recommended actions, leveraging qualys trurisk platform to detect and mitigate the vulnerability, detection and monitoring.

On August 7, 2024, Microsoft disclosed a significant security vulnerability affecting Windows-based systems, known as CVE-2024-21302 . This zero-day vulnerability allows attackers with administrator privileges to elevate their access by replacing current versions of Windows system files with outdated, vulnerable ones.

There are no known exploits in the wild, and Microsoft is unaware of any active exploitation. However, the public disclosure at Black Hat USA 2024 could change the threat landscape, making it imperative for organizations to take preemptive actions. As of the initial publication, exploitation is considered less likely due to the high privileges required and the complexity of the attack. However, vigilance is necessary, given the potential impact.

CVE-2024-21302 affects Windows systems that support Virtualization-Based Security (VBS), including specific Azure Virtual Machine SKUs. This vulnerability could enable attackers to reintroduce previously mitigated vulnerabilities, bypass VBS security features, and exfiltrate sensitive data protected by VBS. Microsoft is actively developing a security update to address this issue but has not yet released it. In the interim, organizations must adopt proactive measures to safeguard their systems.

The vulnerability, identified by a security researcher, specifically impacts Windows 10, Windows 11, Windows Server 2016, and higher versions, including Azure VMs with VBS enabled. The exploit allows an attacker with administrative access to replace current Windows system files with outdated versions, thereby undermining the security provided by VBS.

The vulnerability has a CVSS score of 6.7, categorized as “Important.” Its potential impact includes:

• Confidentiality: High risk of data exfiltration.
• Integrity: There is a high risk of integrity compromise due to reintroducing old vulnerabilities.
• Availability: High risk as critical system files could be tampered with.

Qualys has released the QID 92154 (Microsoft Windows Secure Kernel Mode and Update Stack Elevation of Privilege Vulnerability), starting with vulnsigs version VULNSIGS-2.6.114-2.

This detection logic utilizes WMI (Windows Management Instrumentation) to assess a system’s status of Virtualization-Based Security (VBS). It queries the Win32_DeviceGuard class specifically for the VirtualizationBasedSecurityStatus attribute.

This QID will flag the system if the status is set to 1 (Enabled) or 2 (Enabled and Running), indicating that VBS is enabled on the device. This QID verifies security measures related to device and data integrity through hardware virtualization.

While waiting for Microsoft’s security update, organizations can implement several measures to mitigate the risk:

• Reference: Audit File System – Windows 10
• Reference: Audit Sensitive Privilege Use – Windows 10
• Reference: Identity Protection’s Risk Reports – Azure

While these recommendations do not fully mitigate the vulnerability, they can help reduce the risk of exploitation until the security update is available:

Monitor any access or changes to Windows System Files in real time with Qualys File Integrity Monitoring (FIM)

This vulnerability enables an attacker with administrator privileges on the target system to replace current Windows system files with outdated versions .

Given the nature of this exploit, which involves file replacement, Qualys File Integrity Monitoring (FIM) will detect and respond to this activity in real time. The system will create incidents immediately upon detecting any suspicious activities on the target system. FIM continuously monitors and alerts on attempts to access files, such as handle creation, read/write operations, or modifications to security descriptors.

Setting up the monitoring scope

To monitor changes to system files in real time, you can either create a custom FIM rule or import Qualys’ pre-defined FIM Profile from the Library. The pre-defined profile includes most of the critical system files that need real-time monitoring for any access. You also have the flexibility to customize the policy by adding more files to be monitored, thereby expanding your monitoring scope without affecting the host system.

Where Can I Find System Files on Windows?

The majority of Windows system files are stored in C:\Windows, especially in subfolders like System32 and SysWOW64. But you’ll also find system files scattered throughout user folders (like the appdata folder) and app folders (like ProgramData or the Program Files folders).

Cancelling the Noise: Fine-Tuning Alerts

After selecting all the changes to be monitored, including file access, you will start receiving alerts. The next step involves fine-tuning these alerts by adding inclusion/exclusion filters. This helps in filtering out events from legitimate users and processes, thereby reducing false positives and ensuring that you receive only the events of interest.

Protect Cloud Users:

• Investigate user risk in Azure Active Directory by reviewing Identity Protection’s Risk Reports. Rotate credentials for flagged administrators and enable Multi-Factor Authentication (MFA) to mitigate exposure risks.

Qualys’ Risk Remediation solutions can significantly enhance your security posture against CVE-2024-21302.

Here’s how:

• Provides real-time risk assessment and prioritizes vulnerabilities based on threat intelligence and asset criticality.
• Helps identify and mitigate risks proactively, reducing the window of exposure.
• Automates the deployment of patches once released by Microsoft, ensuring systems are updated promptly.
• Integrates with Qualys Vulnerability Management to provide comprehensive protection against known vulnerabilities.

Microsoft Defender for Endpoint (MDE) has introduced a detection mechanism to alert users of any exploit attempts. Organizations using MDE should integrate and enable this feature for enhanced security monitoring.

• Reference: Integration with Microsoft Defender for Cloud – Microsoft Defender for Endpoint

CVE-2024-21302 poses a critical risk to Windows-based systems, especially those leveraging VBS. While Microsoft develops a security update, organizations must implement recommended actions and leverage tools like Qualys File Integrity Monitoring (FIM) , TruRisk Eliminate, and Patch Management to mitigate potential threats. Stay vigilant and proactive to protect your infrastructure from this evolving vulnerability landscape.

Contributors

• Saeed Abbasi, Product Manager, Vulnerability Research, Qualys
• Lavish Jhamb, Senior Product Manager, Compliance Solutions, Qualys

Comments Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Springer Nature - PMC COVID-19 Collection

Medical image analysis based on deep learning approach

Muralikrishna puttagunta.

Department of Computer Science, School of Engineering and Technology, Pondicherry University, Pondicherry, India

Medical imaging plays a significant role in different clinical applications such as medical procedures used for early detection, monitoring, diagnosis, and treatment evaluation of various medical conditions. Basicsof the principles and implementations of artificial neural networks and deep learning are essential for understanding medical image analysis in computer vision. Deep Learning Approach (DLA) in medical image analysis emerges as a fast-growing research field. DLA has been widely used in medical imaging to detect the presence or absence of the disease. This paper presents the development of artificial neural networks, comprehensive analysis of DLA, which delivers promising medical imaging applications. Most of the DLA implementations concentrate on the X-ray images, computerized tomography, mammography images, and digital histopathology images. It provides a systematic review of the articles for classification, detection, and segmentation of medical images based on DLA. This review guides the researchers to think of appropriate changes in medical image analysis based on DLA.

Introduction

In the health care system, there has been a dramatic increase in demand for medical image services, e.g. Radiography, endoscopy, Computed Tomography (CT), Mammography Images (MG), Ultrasound images, Magnetic Resonance Imaging (MRI), Magnetic Resonance Angiography (MRA), Nuclear medicine imaging, Positron Emission Tomography (PET) and pathological tests. Besides, medical images can often be challenging to analyze and time-consuming process due to the shortage of radiologists.

Artificial Intelligence (AI) can address these problems. Machine Learning (ML) is an application of AI that can be able to function without being specifically programmed, that learn from data and make predictions or decisions based on past data. ML uses three learning approaches, namely, supervised learning, unsupervised learning, and semi-supervised learning. The ML techniques include the extraction of features and the selection of suitable features for a specific problem requires a domain expert. Deep learning (DL) techniques solve the problem of feature selection. DL is one part of ML, and DL can automatically extract essential features from raw input data [ 88 ]. The concept of DL algorithms was introduced from cognitive and information theories. In general, DL has two properties: (1) multiple processing layers that can learn distinct features of data through multiple levels of abstraction, and (2) unsupervised or supervised learning of feature presentations on each layer. A large number of recent review papers have highlighted the capabilities of advanced DLA in the medical field MRI [ 8 ], Radiology [ 96 ], Cardiology [ 11 ], and Neurology [ 155 ].

Different forms of DLA were borrowed from the field of computer vision and applied to specific medical image analysis. Recurrent Neural Networks (RNNs) and convolutional neural networks are examples of supervised DL algorithms. In medical image analysis, unsupervised learning algorithms have also been studied; These include Deep Belief Networks (DBNs), Restricted Boltzmann Machines (RBMs), Autoencoders, and Generative Adversarial Networks (GANs) [ 84 ]. DLA is generally applicable for detecting an abnormality and classify a specific type of disease. When DLA is applied to medical images, Convolutional Neural Networks (CNN) are ideally suited for classification, segmentation, object detection, registration, and other tasks [ 29 , 44 ]. CNN is an artificial visual neural network structure used for medical image pattern recognition based on convolution operation. Deep learning (DL) applications in medical images are visualized in Fig.  1 .

a X-ray image with pulmonary masses [ 121 ] b CT image with lung nodule [ 82 ] c Digitized histo pathological tissue image [ 132 ]

Neural networks

History of neural networks.

The study of artificial neural networks and deep learning derives from the ability to create a computer system that simulates the human brain [ 33 ]. A neurophysiologist, Warren McCulloch, and a mathematician Walter Pitts [ 97 ] developed a primitive neural network based on what has been known as a biological structure in the early 1940s. In 1949, a book titled “Organization of Behavior” [ 100 ] was the first to describe the process of upgrading synaptic weights which is now referred to as the Hebbian Learning Rule. In 1958, Frank Rosenblatt’s [ 127 ] landmark paper defined the structure of the neural network called the perceptron for the binary classification task.

In 1962, Windrow [ 172 ] introduced a device called the Adaptive Linear Neuron (ADALINE) by implementing their designs in hardware. The limitations of perceptions were emphasized by Minski and Papert (1969) [ 98 ]. The concept of the backward propagation of errors for purposes of training is discussed in Werbose1974 [ 171 ]. In 1979, Fukushima [ 38 ] designed artificial neural networks called Neocognitron, with multiple pooling and convolution layers. One of the most important breakthroughs in deep learning occurred in 2006, when Hinton et al. [ 9 ] implemented the Deep Belief Network, with several layers of Restricted Boltzmann Machines, greedily teaching one layer at a time in an unsupervised fashion. In 1989, Yann LeCun [ 71 ] combined CNN with backpropagation to effectively perform the automated recognition of handwritten digits. Figure ​ Figure2 2 shows important advancements in the history of neural networks that led to a deep learning era.

Demonstrations of significant developments in the history of neural networks [ 33 , 134 ]

Artificial neural networks

Artificial Neural Networks (ANN) form the basis for most of the DLA. ANN is a computational model structure that has some performance characteristics similar to biological neural networks. ANN comprises simple processing units called neurons or nodes that are interconnected by weighted links. A biological neuron can be described mathematically in Eq. ( 1 ). Figure ​ Figure3 3 shows the simplest artificial neural model known as the perceptron.

Perceptron [ 77 ]

Training a neural network with Backpropagation (BP)

In the neural networks, the learning process is modeled as an iterative process of optimization of the weights to minimize a loss function. Based on network performance, the weights are modified on a set of examples belonging to the training set. The necessary steps of the training procedure contain forward and backward phases. For Neural Network training, any of the activation functions in forwarding propagation is selected and BP training is used for changing weights. The BP algorithm helps multilayer FFNN to learn input-output mappings from training samples [ 16 ]. Forward propagation and backpropagation are explained with the one hidden layer deep neural networks in the following algorithm.

The backpropagation algorithm is as follows for one hidden layer neural network

• Initialize all weights to small random values.
• While the stopping condition is false, do steps 3 through10.
• For each training pair (( x 1 ,  y 1 )…( x n ,  y n ) do steps 4 through 9.

Feed-forward propagation:

• 4. Each input unit ( X i , i  = 1, 2, … n ) receives the input signal x i and send this signal to all hidden units in the above layer.
• 5. Each hidden unit ( Z j ,  j  = 1. .,  p ) compute output using the below equation, and it transmits to the output unit (i.e.) z j _ in = b j + ∑ i = 1 n w ij x i applies to an activation function Z j  =  f ( Z j  _  in ).

y k _ in = b k + ∑ j = 1 p z j w jk and calculate activation y k  =  f ( y k  _  in )

Backpropagation

At output-layer neurons δ k  = ( t k  −  y k ) f ′ ( y k  _  in )

At Hidden layer neurons δ j = f ′ z j _ in ∑ k m δ k w jk

• 9. Update weights and biases using the following formulas where η is learning rate

Each output layer ( Y k , k  = 1, 2, …. m ) updates its weights ( J  = 0, 1, … P ) and bias

w jk ( new ) =  w jk ( old ) +  ηδ k z j ; b k ( new ) =  b k ( old ) +  ηδ k

Each hidden layer ( Z J ,  J  = 1, 2, … p ) updates its weights ( i  = 0, 1, … n ) biases:

w ij ( new ) =  w ij ( old ) +  ηδ j x i ; b j ( old ) =  b j ( old ) +  ηδ j

• 10. Test stopping condition

Activation function

The activation function is the mechanism by which artificial neurons process and transfers information [ 42 ]. There are various types of activation functions which can be used in neural networks based on the characteristic of the application. The activation functions are non-linear and continuously differentiable. Differentiability property is important mainly when training a neural network using the gradient descent method. Some widely used activation functions are listed in Table ​ Table1 1 .

Activation functions

Deep learning

Deep learning is a subset of the machine learning field which deals with the development of deep neural networks inspired by biological neural networks in the human brain .

Autoencoder

Autoencoder (AE) [ 128 ] is one of the deep learning models which exemplifies the principle of unsupervised representation learning as depicted in Fig.  4a . AE is useful when the input data have more number of unlabelled data compared to labeled data. AE encodes the input x into a lower-dimensional space z. The encoded representation is again decoded to an approximated representation  x ′ of the input x through one hidden layer z.

a Autoencoder [ 187 ] b Restricted Boltzmann Machine with n hidden and m visible units [ 88 ] c Deep Belief Networks [ 88 ]

Basic AE consists of three main steps:

Encode: Convert input vector x ϵ R m into h ϵ R n , the hidden layer by h  =  f ( wx  +  b )where w ϵ R m ∗ n and b ϵ R n . m  and n are dimensions of the input vector and converted hidden state. The dimension of the hidden layer h is to be smaller than x . f is an activate function.

Decode: Based on the above  h , reconstruct input vector z by equation z  =  f ′ ( w ′ h  +  b ′ ) where w ′ ϵ R n ∗ m and b ′ ϵ R m . The f ′ is the same as the above activation function.

Calculate square error: L recons ( x , z) =  ∥  x  − z∥ 2 , which is the reconstruction error cost function. Reconstruct error minimization is achieved by optimizing the cost function (2)

Another unsupervised algorithm representation is known as Stacked Autoencoder (SAE). The SAE comprises stacks of autoencoder layers mounted on top of each other where the output of each layer was wired to the inputs of the next layer. A Denoising Autoencoder (DAE) was introduced by Vincent et al. [ 159 ]. The DAE is trained to reconstruct the input from random noise added input data. Variational autoencoder (VAE) [ 66 ] is modifying the encoder where the latent vector space is used to represent the images that follow a Gaussian distribution unit. There are two losses in this model; one is a mean squared error and the Kull back Leibler divergence loss that determines how close the latent variable matches the Gaussian distribution unit. Sparse autoencoder [ 106 ] and variational autoencoders have applications in unsupervised, semi-supervised learning, and segmentation.

Restricted Boltzmann machine

A Restricted Boltzmann machine [RBM] is a Markov Random Field (MRF) associated with the two-layer undirected probabilistic generative model, as shown in Fig. ​ Fig.4b. 4b . RBM contains visible units (input) v and hidden (output) units  h . A significant feature of this model is that there is no direct contact between the two visible units or either of the two hidden units. In binary RBMs, the random variables ( v ,  h ) takes ( v ,  h ) ∈ {0, 1} m  +  n . Like the general Boltzmann machine [ 50 ], the RBM is an energy-based model. The energy of the state { v ,  h } is defined as (3)

where v j , h i are the binary states of visible unit j  ∈ {1, 2, … m } and hidden unit i  ∈ {1, 2, .. n }, b j , c i  are their biases of visible and hidden units, w ij is the symmetric interaction term between the units v j and h i them. A joint probability of ( v ,  h ) is given by the Gibbs distribution in Eq. ( 4 )

Z is a “partition function” that can be given by summing over all possible pairs of visual v  and hidden h (5).

A significant feature of the RBM model is that there is no direct contact between the two visible units or either of the two hidden units. In term of probability, conditional distributions p ( h |  v ) and p ( v |  h ) is computed as (6) p h v = ∏ i = 1 n p h i v

For binary RBM condition distribution of visible and hidden are given by (7) and (8)

where σ( · ) is a sigmoid function

RBMs parameters ( w ij ,  b j ,  c i ) are efficiently calculated using the contrastive divergence learning method [ 150 ]. A batch version of k-step contrastive divergence learning (CD-k) can be discussed in the algorithm below [ 36 ]

Deep belief networks

The Deep Belief Networks (DBN) proposed by Hinton et al. [ 51 ] is a non-convolution model that can extract features and learn a deep hierarchical representation of training data. DBNs are generative models constructed by stacking multiple RBMs. DBN is a hybrid model, the first two layers are like RBM, and the rest of the layers form a directed generative model. A DBN has one visible layer v and a series of hidden layers h (1) , h (2) , …, h ( l ) as shown in Fig. ​ Fig.4c. 4c . The DBN model joint distribution between the observed units v and the l  hidden layers h k (  k  = 1, … l ) as (9)

where v  =  h (0) , P ( h k |  h k  + 1 ) is a conditional distribution (10) for the layer k given the units of k  + 1

A DBN has l weight matrices: W (1) , …. , W ( l ) and l  + 1 bias vectors: b (0) , …, b ( l ) P ( h ( l ) ,  h ( l  − 1) ) is the joint distribution of top-level RBM (11).

The probability distribution of DBN is given by Eq. ( 12 )

Convolutional neural networks (CNN)

In neural networks, CNN is a unique family of deep learning models. CNN is a major artificial visual network for the identification of medical image patterns. The family of CNN primarily emerges from the information of the animal visual cortex [ 55 , 116 ]. The major problem within a fully connected feed-forward neural network is that even for shallow architectures, the number of neurons may be very high, which makes them impractical to apply to image applications. The CNN is a method for reducing the number of parameters, allows a network to be deeper with fewer parameters.

CNN’s are designed based on three architectural ideas that are shared weights, local receptive fields, and spatial sub-sampling [ 70 ]. The essential element of CNN is the handling of unstructured data through the convolution operation. Convolution of the input signal  x ( t ) with filter signal  h ( t ) creates an output signal y ( t ) that may reveal more information than the input signal itself. 1D convolution of a discrete signals x ( t ) and h ( t ) is (13)

A digital image x ( n 1 ,  n 2 ) is a 2-D discrete signal. The convolution of images  x ( n 1 ,  n 2 ) and h ( n 1 ,  n 2 ) is (14)

where 0 ≤  n 1  ≤  M  − 1, 0 ≤  n 2  ≤  N  − 1.

The function of the convolution layer is to detect local features x l from input feature maps x l  − 1 using kernels k l by convolution operation (*) i.e. x l  − 1  ∗  k l . This convolution operation is repeated for every convolutional layer subject to non-linear transform (15)

where k mn l represents weights between feature map  m at layer l  − 1 and feature map n at l . x m l − 1 represents the  m  feature map of the layer l  − 1 and x n l is n  feature map of the layer l . b m l is the bias parameter. f (.) is the non-linear activation function.  M l  − 1 denotes a set of feature maps. CNN significantly reduces the number of parameters compared with a fully connected neural network because of local connectivity and weight sharing. The depth, zero-padding, and stride are three hyperparameters for controlling the volume of the convolution layer output.

A pooling layer comes after the convolutional layer to subsample the feature maps. The goal of the pooling layers is to achieve spatial invariance by minimizing the spatial dimension of the feature maps for the next convolution layer. Max pooling and average pooling are commonly used two different polling operations to achieve downsampling. Let the size of the pooling region M  and each element in the pooling region is given as x j  = ( x 1 ,  x 2 , … x M  ×  M ), the output after pooling is given as x i . Max pooling and average polling are described in the following Eqs. ( 16 ) and ( 17 ).

The max-pooling method chooses the most superior invariant feature in a pooling region. The average pooling method selects the average of all the features in the pooling area. Thus, the max-pooling method holds texture information that can lead to faster convergence, average pooling method is called Keep background information [ 133 ]. Spatial pyramid pooling [ 48 ], stochastic polling [ 175 ], Def-pooling [ 109 ], Multi activation pooling [ 189 ], and detailed preserving pooling [ 130 ] are different pooling techniques in the literature. A fully connected layer is used at the end of the CNN model. Fully connected layers perform like a traditional neural network [ 174 ]. The input to this layer is a vector of numbers (output of the pooling layer) and outputs an N-dimensional vector (N number of classes). After the pooling layers, the feature of previous layer maps is flattened and connected to fully connected layers.

The first successful seven-layered LeNet-5 CNN was developed by Yann LeCunn in 1990 for handwritten digit recognition successfully. Krizhevsky et al. [ 68 ] proposed AlexNet is a deep convolutional neural network composed of 5 convolutional and 3 fully-connected layers. In AlexNet changed the sigmoid activation function to a ReLU activation function to make model training easier.

K. Simonyan and A. Zisserman invented the VGG-16 [ 143 ] which has 13 convolutional and 3 fully connected layers. The Visual Geometric Group (VGG) research group released a series of CNN starting from VGG-11, VGG-13, VGG-16, and VGG-19. The main intention of the VGG group to understand how the depth of convolutional networks affects the accuracy of the models of image classification and recognition. Compared to the maximum VGG19, which has 16 convolutional layers and 3 fully connected layers, the minimum VGG11 has 8 convolutional layers and 3 fully connected layers. The last three fully connected layers are the same as the various variations of VGG.

Szegedy et al. [ 151 ] proposed an image classification network consisting of 22 different layers, which is GoogleNet. The main idea behind GoogleNet is the introduction of inception layers. Each inception layer convolves the input layers partially using different filter sizes. Kaiming He et al. [ 49 ] proposed the ResNet architecture, which has 33 convolutional layers and one fully-connected layer. Many models introduced the principle of using multiple hidden layers and extremely deep neural networks, but then it was realized that such models suffered from the issue of vanishing or exploding gradients problem. For eliminating vanishing gradients’ problem skip layers (shortcut connections) are introduced. DenseNet developed by Gao et al. [ 54 ] consists of several dense blocks and transition blocks, which are placed between two adjacent dense blocks. The dense block consists of three layers of batch normalization, followed by a ReLU and a 3 × 3 convolution operation. The transition blocks are made of Batch Normalization, 1 × 1 convolution, and average Pooling.

Compared to state-of-the-art handcrafted feature detectors, CNNs is an efficient technique for detecting features of an object and achieving good classification performance. There are drawbacks to CNNs, which are that unique relationships, size, perspective, and orientation of features are not taken into account. To overcome the loss of information in CNNs by pooling operation Capsule Networks (CapsNet) are used to obtain spatial information and most significant features [ 129 ]. The special type of neurons, called capsules, can detect efficiently distinct information. The capsule network consists of four main components that are matrix multiplication, Scalar weighting of the input, dynamic routing algorithm, and squashing function.

Recurrent neural networks (RNN)

RNN is a class of neural networks used for processing sequential information (deal with sequential data). The structure of the RNN shown in Fig.  5a is like an FFNN and the difference is that recurrent connections are introduced among hidden nodes. A generic RNN model at time t , the recurrent connection hidden unit h t receives input activation from the present data x t and the previous hidden state  h t  − 1 . The output y t is calculated given the hidden state h t . It can be represented using the mathematical Eqs. ( 18 ) and ( 19 ) as

a Recurrent Neural Networks [ 163 ] b Long Short-Term Memory [ 163 ] c Generative Adversarial Networks [ 64 ]

Here f is a non-linear activation function, w hx is the weight matrix between the input and hidden layers, w hh is the matrix of recurrent weights between the hidden layers and itself w yh is the weight matrix between the hidden and output layer, and b h and b y are biases that allow each node to learn and offset. While the RNN is a simple and efficient model, in reality, it is, unfortunately, difficult to train properly. Real-Time Recurrent Learning (RTRL) algorithm [ 173 ] and Back Propagation Through Time (BPTT) [ 170 ] methods are used to train RNN. Training with these methods frequently fails because of vanishing (multiplication of many small values) or explode (multiplication of many large values) gradient problem [ 10 , 112 ]. Hochreiter and Schmidhuber (1997) designed a new RNN model named Long Short Term Memory (LSTM) that overcome error backflow problems with the aid of a specially designed memory cell [ 52 ]. Figure ​ Figure5b 5b shows an LSTM cell which is typically configured by three gates: input gate g t , forget gate  f t and output gate  o t , these gates add or remove information from the cell.

An LSTM can be represented with the following Eqs. ( 20 ) to ( 25 )

Generative adversarial networks (GAN)

In the field of deep learning, one of the deep generative models are Generative Adversarial Networks (GANs) introduced by Good Fellow in [ 43 ]. GANs are neural networks that can generate synthetic images that closely imitate the original images. In GAN shown in Fig. ​ Fig.5c, 5c , there are two neural networks, namely generator, and discriminator, which are trained simultaneously. The generator G generates counterfeit data samples which aim to “fool” the discriminator  D , while the discriminator attempts to correctly distinguish the true and false samples. In mathematical terms, D and G play a two player minimax game with the cost function of (26) [ 64 ].

Where x represents the original image, z is a noise vector with random numbers. p data ( x ) and p z ( z ) are probability distributions of x and  z , respectively.  D ( x ) represents the probability that x comes from the actual data p data ( x ) rather than the generated data. 1 −  D ( G (z)) is the probability that it can be generated from p z (z). The expectation of x from the real data distribution  p data is expressed by E x ~ p data x and the expectation of z sampled from noise is E z ~ P z z . The goal of the training is to maximize the loss function for the discriminator, while the training objective for the generator is to reduce the term log (1 −  D ( G ( z ))).The most utilization of GAN in the field of medical image analysis is data augmentation (generating new data) and image to image translation [ 107 ]. Trustability of the Generated Data, Unstable Training, and evaluation of generated data are three major drawbacks of GAN that might hinder their acceptance in the medical community [ 183 ].

Ronneberger et al. [ 126 ] proposed CNN based U-Net architecture for segmentation in biomedical image data. The architecture consists of a contracting path (left side) to capture context and an expansive symmetric path (right side) that enables precise localization. U-Net is a generalized DLA used for quantification tasks such as cell detection and shape measurement in medical image data [ 34 ].

Software frameworks

There are several software frameworks available for implementing DLA which are regularly updated as new approaches and ideas are created. DLA encapsulates many levels of mathematical principles based on probability, linear algebra, calculus, and numerical computation. Several deep learning frameworks exist such as Theano, TensorFlow, Caffe, CNTK, Torch, Neon, pylearn, etc. [ 138 ]. Globally, Python is probably the most commonly used programming language for DL. PyTorch and Tensorflow are the most widely used libraries for research in 2019. Table ​ Table2 2 shows the analysis of various Deep Learning Frameworks based on the core language and supported interface language.

Comparison of various Deep Learning Frameworks

Use of deep learning in medical imaging

X-ray image.

Chest radiography is widely used in diagnosis to detect heart pathologies and lung diseases such as tuberculosis, atelectasis, consolidation, pleural effusion, pneumothorax, and hyper cardiac inflation. X-ray images are accessible, affordable, and less dose-effective compared to other imaging methods, and it is a powerful tool for mass screening [ 14 ]. Table ​ Table3 3 presents a description of the DL methods used for X-ray image analysis.

An overview of the DLA for the study of X-ray images

S. Hwang et al. [ 57 ] proposed the first deep CNN-based Tuberculosis screening system with a transfer learning technique. Rajaraman et al. [ 119 ] proposed modality-specific ensemble learning for the detection of abnormalities in chest X-rays (CXRs). These model predictions are combined using various ensemble techniques toward minimizing prediction variance. Class selective mapping of interest (CRM) is used for visualizing the abnormal regions in the CXR images. Loey et al. [ 90 ] proposed A GAN with deep transfer training for COVID-19 detection in CXR images. The GAN network was used to generate more CXR images due to the lack of the COVID-19 dataset. Waheed et al. [ 160 ] proposed a CovidGAN model based on the Auxiliary Classifier Generative Adversarial Network (ACGAN) to produce synthetic CXR images for COVID-19 detection. S. Rajaraman and S. Antani [ 120 ] introduced weakly labeled data augmentation for increasing training dataset to improve the COVID-19 detection performance in CXR images.

Computerized tomography (CT)

CT uses computers and rotary X-ray equipment to create cross-section images of the body. CT scans show the soft tissues, blood vessels, and bones in different parts of the body. CT is a high detection ability, reveals small lesions, and provides a more detailed assessment. CT examinations are frequently used for pulmonary nodule identification [ 93 ]. The detection of malignant pulmonary nodules is fundamental to the early diagnosis of lung cancer [ 102 , 142 ]. Table ​ Table4 4 summarizes the latest deep learning developments in the study of CT image analysis.

A review of articles that use DL techniques for the analysis of the CT image

AUC: area under ROC curve; FROC: Area under the Free-Response ROC Curve; SN: sensitivity; SP: specificity; MAE: mean absolute error LIDC: Lung Image Database Consortium; LIDC-IDRI: Lung Image Database Consortium-Image Database Resource Initiative.

Li et al. 2016 [ 74 ] proposed deep CNN for the detection of three types of nodules that are semisolid, solid, and ground-glass opacity. Balagourouchetty et al. [ 5 ] proposed GoogLeNet based an ensemble FCNet classifier for The liver lesion classification. For feature extraction, basic Googlenet architecture is modified with three modifications. Masood et al. [ 95 ] proposed the multidimensional Region-based Fully Convolutional Network (mRFCN) for lung nodule detection/classification and achieved a classification accuracy of 97.91%. In lung nodule detection, the feature work is the detection of micronodules (less than 3 mm) without loss of sensitivity and accuracy. Zhao and Zeng 2019 [ 190 ] proposed DLA based on supervised MSS U-Net and 3DU-Net to automatically segment kidneys and kidney tumors from CT images. In the present pandemic situation, Fan et al. [ 35 ] and Li et al. [ 79 ] used deep learning-based techniques for COVID-19 detection from CT images.

Mammograph (MG)

Breast cancer is one of the world’s leading causes of death among women with cancer. MG is a reliable tool and the most common modality for early detection of breast cancer. MG is a low-dose x-ray imaging method used to visualize the breast structure for the detection of breast diseases [ 40 ]. Detection of breast cancer on mammography screening is a difficult task in image classification because the tumors constitute a small part of the actual breast image. For analyzing breast lesions from MG, three steps are involved that are detection, segmentation, and classification [ 139 ].

The automatic classification and detection of masses at an early stage in MG is still a hot subject of research. Over the past decade, DLA has shown some significant overcome in breast cancer detection and classification problem. Table ​ Table5 5 summarizes the latest DLA developments in the study of mammogram image analysis.

Summary of DLA for MG image analysis

MIAS: Mammographic Image Analysis Society dataset; DDSM: Digital Database for Screening Mammography; BI-RADS: Breast Imaging Reporting and Data System; `WBCD: Wisconsin Breast Cancer Dataset; DIB-MG: data-driven imaging biomarker in mammography. FFDMs: Full-Field Digital Mammograms; MAMMO: Man and Machine Mammography Oracle; FROC: Free response receiver operating characteristic analysis; SN: sensitivity; SP: specificity.

Fonseca et al. [ 37 ] proposed a breast composition classification according to the ACR standard based on CNN for feature extraction. Wang et al. [ 161 ] proposed twelve-layer CNN to detect Breast arterial calcifications (BACs) in mammograms image for risk assessment of coronary artery disease. Ribli et al. [ 124 ] developed a CAD system based on Faster R-CNN for detection and classification of benign and malignant lesions on a mammogram image without any human involvement. Wu et al. [ 176 ] present a deep CNN trained and evaluated on over 1,000,000 mammogram images for breast cancer screening exam classification. Conant et al. [ 26 ] developed a Deep CNN based AI system to detect calcified lesions and soft- tissue in digital breast tomosynthesis (DBT) images. Kang et al. [ 62 ] introduced Fuzzy completely connected layer (FFCL) architecture, which focused primarily on fused fuzzy rules with traditional CNN for semantic BI-RADS scoring. The proposed FFCL framework achieved superior results in BI-RADS scoring for both triple and multi-class classifications.

Histopathology

Histopathology is the field of study of human tissue in the sliding glass using a microscope to identify different diseases such as kidney cancer, lung cancer, breast cancer, and so on. The staining is used in histopathology for visualization and highlight a specific part of the tissue [ 45 ]. For example, Hematoxylin and Eosin (H&E) staining tissue gives a dark purple color to the nucleus and pink color to other structures. H&E stain plays a key role in the diagnosis of different pathologies, cancer diagnosis, and grading over the last century. The recent imaging modality is digital pathology

Deep learning is emerging as an effective method in the analysis of histopathology images, including nucleus detection, image classification, cell segmentation, tissue segmentation, etc. [ 178 ]. Tables ​ Tables6 6 and ​ and7 7 summarize the latest deep learning developments in pathology. In the study of digital pathology image analysis, the latest development is the introduction of whole slide imaging (WSI). WSI allows digitizing glass slides with stained tissue sections at high resolution. Dimitriou et al. [ 30 ] reviewed challenges for the analysis of multi-gigabyte WSI images for building deep learning models. A. Serag et al. [ 135 ] discuss different public “Grand Challenges” that have innovations using DLA in computational pathology.

Summary of articles using DLA for digital pathology image - Organ segmentation

Summary of articles using DLA for digital pathology image - Detection and classification of disease

NODE: Neural Ordinary Differential Equations; IoU: mean Intersection over Union coefficient

Other images

Endoscopy is the insertion of a long nonsurgical solid tube directly into the body for the visual examination of an internal organ or tissue in detail. Endoscopy is beneficial in studying several systems inside the human body, such as the gastrointestinal tract, the respiratory tract, the urinary tract, and the female reproductive tract [ 60 , 101 ]. Du et al. [ 31 ] reviewed the Applications of Deep Learning in the Analysis of Gastrointestinal Endoscopy Images. A revolutionary device for direct, painless, and non-invasive inspection of the gastrointestinal (GI) tract for detecting and diagnosing GI diseases (ulcer, bleeding) is Wireless capsule endoscopy (WCE). Soffer et al. [ 145 ] performed a systematic analysis of the existing literature on the implementation of deep learning in the WCE. The first deep learning-based framework was proposed by He et al. [ 46 ] for the detection of hookworm in WCE images. Two CNN networks integrated (edge extraction and classification of hookworm) to detect hookworm. Since tubular structures are crucial elements for hookworm detection, the edge extraction network was used for tubular region detection. Yoon et al. [ 185 ] developed a CNN model for early gastric cancer (EGC) identification and prediction of invasion depth. The depth of tumor invasion in early gastric cancer (EGC) is a significant factor in deciding the method of treatment. For the classification of endoscopic images as EGC or non-EGC, the authors employed a VGG-16 model. Nakagawa et al. [ 105 ] applied DL technique based on CNN to enhance the diagnostic assessment of oesophageal wall invasion using endoscopy. J.choi et al. [ 22 ] express the feature aspects of DL in endoscopy.

Positron Emission Tomography (PET) is a nuclear imaging tool that is generally used by the injection of particular radioactive tracers to visualize molecular-level activities within tissues. T. Wang et al. [ 168 ] reviewed applications of machine learning in PET attenuation correction (PET AC) and low-count PET reconstruction. The authors discussed the advantages of deep learning over machine learning in the applications of PET images. AJ reader et al. [ 123 ] reviewed the reconstruction of PET images that can be used in deep learning either directly or as a part of traditional reconstruction methods.

The primary purpose of this paper is to review numerous publications in the field of deep learning applications in medical images. Classification, detection, and segmentation are essential tasks in medical image processing [ 144 ]. For specific deep learning tasks in medical applications, the training of deep neural networks needs a lot of labeled data. But in the medical field, at least thousands of labeled data is not available. This issue is alleviated by a technique called transfer learning. Two transfer learning approaches are popular and widely applied that are fixed feature extractors and fine-tuning a pre-trained network. In the classification process, the deep learning models are used to classify images into two or more classes. In the detection process, Deep learning models have the function of identifying tumors and organs in medical images. In the segmentation task, deep learning models try to segment the region of interest in medical images for processing.

Segmentation

For medical image segmentation, deep learning has been widely used, and several articles have been published documenting the progress of deep learning in the area. Segmentation of breast tissue using deep learning alone has been successfully implemented [ 104 ]. Xing et al. [ 179 ] used CNN to acquire the initial shape of the nucleus and then isolate the actual nucleus using a deformable pattern. Qu et al. [ 118 ] suggested a deep learning approach that could segment the individual nucleus and classify it as a tumor, lymphocyte, and stroma nuclei. Pinckaers and Litjens [ 115 ] show on a colon gland segmentation dataset (GlaS) that these Neural Ordinary Differential Equations (NODE) can be used within the U-Net framework to get better segmentation results. Sun 2019 [ 149 ] developed a deep learning architecture for gastric cancer segmentation that shows the advantage of utilizing multi-scale modules and specific convolution operations together. Figure ​ Figure6 6 shows U-Net is the most usually used network for segmentation (Fig. ​ (Fig.6 6 ).

U-Net architecture for segmentation,comprising encoder (downsampling) and decoder (upsampling) sections [ 135 ]

The main challenge posed by methods of detection of lesions is that they can give rise to multiple false positives while lacking a good proportion of true positive ones . For tuberculosis detection using deep learning methods applied in [ 53 , 57 , 58 , 91 , 119 ]. Pulmonary nodule detection using deep learning has been successfully applied in [ 82 , 108 , 136 , 157 ].

Shin et al. [ 141 ] discussed the effect of CNN pre-trained architectures and transfer learning on the identification of enlarged thoracoabdominal lymph nodes and the diagnosis of interstitial lung disease on CT scans, and considered transfer learning to be helpful, given the fact that natural images vary from medical images. Litjens et al. [ 85 ] introduced CNN for the identification of Prostate cancer in biopsy specimens and breast cancer metastasis identification in sentinel lymph nodes. The CNN has four convolution layers for feature extraction and three classification layers. Riddle et al. [ 124 ] proposed the Faster R-CNN model for the detection of mammography lesions and classified these lesions into benign and malignant, which finished second in the Digital Mammography DREAM Challenge. Figure ​ Figure7 7 shows VGG architecture for detection.

CNN architecture for detection [ 144 ]

An object detection framework named Clustering CNN (CLU-CNNs) was proposed by Z. Li et al. [ 76 ] for medical images. CLU-CNNs used Agglomerative Nesting Clustering Filtering (ANCF) and BN-IN Net to avoid much computation cost facing medical images. Image saliency detection aims at locating the most eye-catching regions in a given scene [ 21 , 78 ]. The goal of image saliency detection is to locate a given scene in the most eye-catching regions. In different applications, it also acts as a pre-processing tool including video saliency detection [ 17 , 18 ], object recognition, and object tracking [ 20 ]. Saliency maps are a commonly used tool for determining which areas are most important to the prediction of a trained CNN on the input image [ 92 ]. NT Arun et al. [ 4 ] evaluated the performance of several popular saliency methods on the RSNA Pneumonia Detection dataset and was found that GradCAM was sensitive to the model parameters and model architecture.

Classification

In classification tasks, deep learning techniques based on CNN have seen several advancements. The success of CNN in image classification has led researchers to investigate its usefulness as a diagnostic method for identifying and characterizing pulmonary nodules in CT images. The classification of lung nodules using deep learning [ 74 , 108 , 117 , 141 ] has also been successfully implemented.

Breast parenchymal density is an important indicator of the risk of breast cancer. The DL algorithms used for density assessment can significantly reduce the burden of the radiologist. Breast density classification using DL has been successfully implemented [ 37 , 59 , 72 , 177 ]. Ionescu et al. [ 59 ] introduced a CNN-based method to predict Visual Analog Score (VAS) for breast density estimation. Figure ​ Figure8 8 shows AlexNet architecture for classification.

CNN architecture for classification [ 144 ]

Alcoholism or alcohol use disorder (AUD) has effects on the brain. The structure of the brain was observed using the Neuroimaging approach. S.H.Wang et al. [ 162 ] proposed a 10-layer CNN for alcohol use disorder (AUD) problem using dropout, batch normalization, and PReLU techniques. The authors proposed a 10 layer CNN model that has obtained a sensitivity of 97.73, a specificity of 97.69, and an accuracy of 97.71. Cerebral micro-bleeding (CMB) are small chronic brain hemorrhages that can result in cognitive impairment, long-term disability, and neurologic dysfunction. Therefore, early-stage identification of CMBs for prompt treatment is essential. S. Wang et al. [ 164 ] proposed the transfer learning-based DenseNet to detect Cerebral micro-bleedings (CMBs). DenseNet based model attained an accuracy of 97.71% (Fig. ​ (Fig.8 8 ).

Limitations and challenges

The application of deep learning algorithms to medical imaging is fascinating, but many challenges are pulling down the progress. One of the limitations to the adoption of DL in medical image analysis is the inconsistency in the data itself (resolution, contrast, signal-to-noise), typically caused by procedures in clinical practice [ 113 ]. The non-standardized acquisition of medical images is another limitation in medical image analysis. The need for comprehensive medical image annotations limits the applicability of deep learning in medical image analysis. The major challenge is limited data and compared to other datasets, the sharing of medical data is incredibly complicated. Medical data privacy is both a sociological and a technological issue that needs to be discussed from both viewpoints. For building DLA a large amount of annotated data is required. Annotating medical images is another major challenge. Labeling medical images require radiologists’ domain knowledge. Therefore, it is time-consuming to annotate adequate medical data. Semi-supervised learning could be implemented to make combined use of the existing labeled data and vast unlabelled data to alleviate the issue of “limited labeled data”. Another way to resolve the issue of “data scarcity” is to develop few-shot learning algorithms using a considerably smaller amount of data. Despite the successes of DL technology, there are many restrictions and obstacles in the medical field. Whether it is possible to reduce medical costs, increase medical efficiency, and improve the satisfaction of patients using DL in the medical field cannot be adequately checked. However, in clinical trials, it is necessary to demonstrate the efficacy of deep learning methods and to develop guidelines for the medical image analysis applications of deep learning.

Conclusion and future directions

Medical imaging is a place of origin of the information necessary for clinical decisions. This paper discusses the new algorithms and strategies in the area of deep learning. In this brief introduction to DLA in medical image analysis, there are two objectives. The first one is an introduction to the field of deep learning and the associated theory. The second is to provide a general overview of the medical image analysis using DLA. It began with the history of neural networks since 1940 and ended with breakthroughs in medical applications in recent DL algorithms. Several supervised and unsupervised DL algorithms are first discussed, including auto-encoders, recurrent, CNN, and restricted Boltzmann machines. Several optimization techniques and frameworks in this area include Caffe, TensorFlow, Theano, and PyTorch are discussed. After that, the most successful DL methods were reviewed in various medical image applications, including classification, detection, and segmentation. Applications of the RBM network is rarely published in the medical image analysis literature. In classification and detection, CNN-based models have achieved good results and are most commonly used. Several existing solutions to medical challenges are available. However, there are still several issues in medical image processing that need to be addressed with deep learning. Many of the current DL implementations are supervised algorithms, while deep learning is slowly moving to unsupervised and semi-supervised learning to manage real-world data without manual human labels.

DLA can support clinical decisions for next-generation radiologists. DLA can automate radiologist workflow and facilitate decision-making for inexperienced radiologists. DLA is intended to aid physicians by automatically identifying and classifying lesions to provide a more precise diagnosis. DLA can help physicians to minimize medical errors and increase medical efficiency in the processing of medical image analysis. DL-based automated diagnostic results using medical images for patient treatment are widely used in the next few decades. Therefore, physicians and scientists should seek the best ways to provide better care to the patient with the help of DLA. The potential future research for medical image analysis is the designing of deep neural network architectures using deep learning. The enhancement of the design of network structures has a direct impact on medical image analysis. Manual design of DL Model structure requires rich knowledge; hence Neural Network Search will probably replace the manual design [ 73 ]. A meaningful feature research direction is also the design of various activation functions. Radiation therapy is crucial for cancer treatment. Different medical imaging modalities are playing a critical role in treatment planning. Radiomics was defined as the extraction of high throughput features from medical images [ 28 ]. In the feature, Deep-learning analysis of radionics will be a promising tool in clinical research for clinical diagnosis, drug development, and treatment selection for cancer patients . Due to limited annotated medical data, unsupervised, weakly supervised, and reinforcement learning methods are the emerging research areas in DL for medical image analysis. Overall, deep learning, a new and fast-growing field, offers various obstacles as well as opportunities and solutions for a range of medical image applications.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Muralikrishna Puttagunta, Email: moc.liamg@04939ilarum .

S. Ravi, Email: moc.liamg@eticivars .

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

A study on the evolution of forest landscape patterns in the fuxin region of china combining sc-unet and spatial pattern perspectives.

1. Introduction

2. study region and data, 2.1. overview of the study region, 2.2. research information, 3. research methodology, 3.1. spatial and channel reconstruction convolution, 3.2. sc-unet forest extraction model, 3.3. a morphology-based approach to analyzing forest spatial patterns, 3.4. multivariate weighted results, 4. experiments and analysis, 4.1. experimental data, 4.2. model training, 4.3. experimental results, 5. discussion, 5.1. spatial and temporal changes of forest land in the fuxin region, 5.2. evolution of the spatio-temporal patterns of forest landscape categories in the fuxin region based on mspa, 5.3. limitations and future research, 6. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, acknowledgments, conflicts of interest.

• Peng, X.; He, G.; She, W.; Zhang, X.; Wang, G.; Yin, R.; Long, T. A Comparison of Random Forest Algorithm-Based Forest Extraction with GF-1 WFV, Landsat 8 and Sentinel-2 Images. Remote Sens. 2022 , 14 , 5296. [ Google Scholar ] [ CrossRef ]
• Zhu, H.; Zhang, B.; Song, W.; Xie, Q.; Chang, X.; Zhao, R. Forest Canopy Height Estimation by Integrating Structural Equation Modeling and Multiple Weighted Regression. Forests 2024 , 15 , 369. [ Google Scholar ] [ CrossRef ]
• Zhang, B.; Zhu, H.; Xu, W.; Xu, S.; Chang, X.; Song, W.; Zhu, J. A Fourier–Legendre Polynomial Forest Height Inversion Model Based on a Single-Baseline Configuration. Forests 2024 , 15 , 49. [ Google Scholar ] [ CrossRef ]
• De Frenne, P.; Lenoir, J.; Luoto, M.; Scheffers, B.; Zellweger, F.; Aalto, J.; Ashcroft, M.; Christiansen, D.; Decocq, G.; De Pauw, K.; et al. Forest microclimates and climate change: Importance, drivers and future research agenda. Glob. Chang. Biol. 2021 , 27 , 2279–2297. [ Google Scholar ] [ CrossRef ]
• Ripple, W.J.; Bradshaw, G.A.; Spies, T.A. Measuring forest landscape patterns in the Cascade Range of Oregon, USA. Biol. Conserv. 1991 , 57 , 73–88. [ Google Scholar ] [ CrossRef ]
• Wang, J.; Yang, X. A hierarchical approach to forest landscape pattern characterization. Environ. Manag. 2012 , 49 , 64–81. [ Google Scholar ] [ CrossRef ]
• Tariq, A.; Jiango, Y.; Li, Q.; Gao, J.; Lu, L.; Soufan, W.; Almutairi, K.; Habib-ur-Rahman, M. Modelling, mapping and monitoring of forest cover changes, using support vector machine, kernel logistic regression and naive bayes tree models with optical remote sensing data. Heliyon 2023 , 9 , e13212. [ Google Scholar ] [ CrossRef ]
• Cross, M.D.; Scambos, T.; Pacifici, F.; Marshall, W.E. Determining effective meter-scale image data and spectral vegetation indices for tropical forest tree species differentiation. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2019 , 12 , 2934–2943. [ Google Scholar ] [ CrossRef ]
• Zhang, X.; Zhang, Y.; Liu, L.; Zhang, J.; Gao, J. Remote sensing monitoring of the subalpine coniferous forests and quantitative analysis of the characteristics of succession in east mountain area of Tibetan Plateau—A case study with Zamtange county. Agric. Sci. Technol. 2011 , 12 , 926–930. [ Google Scholar ]
• Franklin, S.E.; Hall, R.J.; Moskal, L.M.; Maudie, A.J.; Lavigne, M.B. Incorporating texture into classification of forest species composition from airborne multispectral images. Int. J. Remote Sens. 2000 , 21 , 61–79. [ Google Scholar ] [ CrossRef ]
• Cheng, K.; Wang, J. Forest type classification based on integrated spectral-spatial-temporal features and random forest algorithm—A case study in the Qinling mountains. Forests 2019 , 10 , 559. [ Google Scholar ] [ CrossRef ]
• Xie, G.; Niculescu, S. Mapping and monitoring of land cover/land use (LCLU) changes in the Crozon peninsula (Brittany, France) from 2007 to 2018 by machine learning algorithms (support vector machine, random forest, and convolutional neural network) and by post-classification comparison (PCC). Remote Sens. 2021 , 13 , 3899. [ Google Scholar ] [ CrossRef ]
• Haq, M.A.; Rahaman, G.; Baral, P.; Ghosh, A. Deep learning based supervised image classification using UAV images for forest areas classification. J. Indian Soc. Remote Sens. 2021 , 49 , 601–606. [ Google Scholar ] [ CrossRef ]
• Chai, J.; Zeng, H.; Li, A.; Ngai, E.W.T. Deep learning in computer vision: A critical review of emerging techniques and application scenarios. Mach. Learn. Appl. 2021 , 6 , 100134. [ Google Scholar ] [ CrossRef ]
• Dhanya, V.G.; Subeesh, A.; Kushwaha, N.L.; Vishwakarma, D.K.; Kumar, T.N.; Ritika, G.; Singh, A.N. Deep learning based computer vision approaches for smart agricultural applications. Artif. Intell. Agric. 2022 , 6 , 211–229. [ Google Scholar ] [ CrossRef ]
• Moshayedi, A.J.; Roy, A.S.; Kolahdooz, A.; Shuxin, Y. Deep learning application pros and cons over algorithm deep learning application pros and cons over algorithm. EAI Endorsed Trans. AI Robot. 2022 , 1 , 7. [ Google Scholar ]
• Ghasemi, Y.; Jeong, H.; Choi, S.H.; Park, K.B.; Lee, J.Y. Deep learning-based object detection in augmented reality: A systematic review. Comput. Ind. 2022 , 139 , 103661. [ Google Scholar ] [ CrossRef ]
• Chang, X.; Zhang, B.; Zhu, H.; Song, W.; Ren, D.; Dai, J. A Spatial and Temporal Evolution Analysis of Desert Land Changes in Inner Mongolia by Combining a Structural Equation Model and Deep Learning. Remote Sens. 2023 , 15 , 3617. [ Google Scholar ] [ CrossRef ]
• Dong, S.; Wang, P.; Abbas, K. A survey on deep learning and its applications. Comput. Sci. Rev. 2021 , 40 , 100379. [ Google Scholar ] [ CrossRef ]
• Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3431–3440. [ Google Scholar ]
• Wang, J.; Song, L.; Li, Z.; Sun, H.; Sun, J.; Zheng, N. End-to-end object detection with fully convolutional network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 15849–15858. [ Google Scholar ]
• Wu, S.; Shi, J.; Chen, Z. HG-FCN: Hierarchical grid fully convolutional network for fast VVC intra coding. IEEE Trans. Circuits Syst. Video Technol. 2022 , 32 , 5638–5649. [ Google Scholar ] [ CrossRef ]
• Wang, S.; Liu, C.; Zhang, Y. Fully convolution network architecture for steel-beam crack detection in fast-stitching images. Mech. Syst. Signal Process. 2022 , 165 , 108377. [ Google Scholar ] [ CrossRef ]
• Zhu, H.; Zhang, B.; Chang, X.; Song, W.; Dai, J.; Li, J. A Study of Sandy Land Changes in the Chifeng Region from 1990 to 2020 Based on Dynamic Convolution. Sustainability 2023 , 15 , 12931. [ Google Scholar ] [ CrossRef ]
• Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, 5–9 October 2015; proceedings, part III 18. Springer International Publishing: Berlin/Heidelberg, Germany, 2015; pp. 234–241. [ Google Scholar ]
• Su, Z.; Li, W.; Ma, Z.; Gao, R. An improved U-Net method for the semantic segmentation of remote sensing images. Appl. Intell. 2022 , 52 , 3276–3288. [ Google Scholar ] [ CrossRef ]
• John, D.; Zhang, C. An attention-based U-Net for detecting deforestation within satellite sensor imagery. Int. J. Appl. Earth Obs. Geoinf. 2022 , 107 , 102685. [ Google Scholar ] [ CrossRef ]
• Meena, S.R.; Soares, L.P.; Grohmann, C.H.; Westen, C.; Bhuyan, K.; Singh, R.P.; Floris, M.; Catani, F. Landslide detection in the Himalayas using machine learning algorithms and U-Net. Landslides 2022 , 19 , 1209–1229. [ Google Scholar ] [ CrossRef ]
• Alsabhan, W.; Alotaiby, T.; Dudin, B. Detecting buildings and nonbuildings from satellite images using U-Net. Comput. Intell. Neurosci. 2022 , 2022 , 4831223. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Wang, H.; Miao, F. Building extraction from remote sensing images using deep residual U-Net. Eur. J. Remote Sens. 2022 , 55 , 71–85. [ Google Scholar ] [ CrossRef ]
• Yan, C.; Fan, X.; Fan, J.; Wang, N. Improved U-Net remote sensing classification algorithm based on Multi-Feature Fusion Perception. Remote Sens. 2022 , 14 , 1118. [ Google Scholar ] [ CrossRef ]
• Zhang, R.; Zhu, D. Study of land cover classification based on knowledge rules using high-resolution remote sensing images. Expert Syst. Appl. 2011 , 38 , 3647–3652. [ Google Scholar ] [ CrossRef ]
• Chen, B.; Xia, M.; Huang, J. Mfanet: A multi-level feature aggregation network for semantic segmentation of land cover. Remote Sens. 2021 , 13 , 731. [ Google Scholar ] [ CrossRef ]
• Navin, M.S.; Agilandeeswari, L. Comprehensive review on land use/land cover change classification in remote sensing. J. Spectr. Imaging 2020 , 9 , a8. [ Google Scholar ] [ CrossRef ]
• Kupidura, P. The Comparison of Different Methods of Texture Analysis for Their Efficacy for Land Use Classification in Satellite Imagery. Remote Sens. 2019 , 11 , 1233. [ Google Scholar ] [ CrossRef ]
• Liu, M.; Shi, J.; Li, Z.; Li, C.; Zhu, J.; Liu, S. Towards better analysis of deep convolutional neural networks. IEEE Trans. Vis. Comput. Graph. 2016 , 23 , 91–100. [ Google Scholar ] [ CrossRef ]
• Xu, R.; Samat, A.; Zhu, E.; Li, E.; Li, W. Unsupervised Domain Adaptation with Contrastive Learning-Based Discriminative Feature Augmentation for RS Image Classification. Remote Sens. 2024 , 16 , 1974. [ Google Scholar ] [ CrossRef ]
• Lu, Y.; Li, H.; Zhang, C.; Zhang, S. Object-Based Semi-Supervised Spatial Attention Residual UNet for Urban High-Resolution Remote Sensing Image Classification. Remote Sens. 2024 , 16 , 1444. [ Google Scholar ] [ CrossRef ]
• Han, Y.; Guo, J.; Yang, H.; Guan, R.; Zhang, T. SSMA-YOLO: A Lightweight YOLO Model with Enhanced Feature Extraction and Fusion Capabilities for Drone-Aerial Ship Image Detection. Drones 2024 , 8 , 145. [ Google Scholar ] [ CrossRef ]
• Wu, Q.; Feng, D.; Cao, C.; Zeng, X.; Feng, Z.; Wu, J.; Huang, Z. Improved Mask R-CNN for Aircraft Detection in Remote Sensing Images. Sensors 2021 , 21 , 2618. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Li, S.; Ganguly, S.; Dungan, J.L.; Wang, W.; Nemani, R.R. Sentinel-2 MSI radiometric characterization and cross-calibration with Landsat-8 OLI. Adv. Remote Sens. 2017 , 6 , 147. [ Google Scholar ] [ CrossRef ]
• Gašparović, M.; Jogun, T. The effect of fusing Sentinel-2 bands on land-cover classification. Int. J. Remote Sens. 2018 , 39 , 822–841. [ Google Scholar ] [ CrossRef ]
• Lamquin, N.; Woolliams, E.; Bruniquel, V.; Gascon, F.; Gorroño, J.; Govaerts, Y.; Leroy, V.; Lonjou, V.; Alhammoud, B.; Barsi, J.A.; et al. An inter-comparison exercise of Sentinel-2 radiometric validations assessed by independent expert groups. Remote Sens. Environ. 2019 , 233 , 111369. [ Google Scholar ] [ CrossRef ]
• Su, W.; Zhang, M.; Jiang, K.; Zhu, D.; Huang, J.; Wang, P. Atmospheric correction method for Sentinel-2 satellite imagery. Acta Opt. Sin. 2018 , 38 , 0128001. [ Google Scholar ] [ CrossRef ]
• Yin, F.; Lewis, P.E.; Gómez-Dans, J.L. Bayesian atmospheric correction over land: Sentinel-2/MSI and Landsat 8/OLI. Geosci. Model Dev. 2022 , 15 , 7933–7976. [ Google Scholar ] [ CrossRef ]
• Li, J.F.; Wen, Y.; He, L.H. SCConv: Spatial and channel reconstruction convolution for feature redundancy. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; IEEE Press: New York, NY, USA, 2023; pp. 6153–6162. [ Google Scholar ]
• Chen, C.F.; Oh, J.; Fan, Q.; Pistoia, M. SC-Conv: Sparse-complementary convolution for efficient model utilization on CNNs. In Proceedings of the 2018 IEEE International Symposium on Multimedia (ISM), Taichung, Taiwan, 10–12 December 2018; pp. 97–100. [ Google Scholar ]
• Wu, Y.; He, K. Group Normalization. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; Volume 3, pp. 3–19. [ Google Scholar ]
• Li, X.; Wang, W.; Hu, X.; Yang, J. Selective kernel networks. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; IEEE Press: New York, NY, USA, 2019; pp. 510–519. [ Google Scholar ]
• Ogundokun, R.O.; Maskeliunas, R.; Misra, S.; Damaševičius, R. Improved CNN based on batch normalization and Adam optimizer. In Proceedings of the International Conference on Computational Science and Its Applications, Malaga, Spain, 4–7 July 2022; Springer International Publishing: Cham, Switzerland, 2022; pp. 593–604. [ Google Scholar ]
• Lange, S.; Helfrich, K.; Ye, Q. Batch normalization preconditioning for neural network training. J. Mach. Learn. Res. 2022 , 23 , 1–41. [ Google Scholar ]
• Yu, X.; Zheng, Z.; Meng, L.; Li, L. Scene Recognition of Remotely Sensed Images Based on Bayes Adjoint Batch Normalization ; Geomatics and Information Science of Wuhan University: Wuhan, China, 2023. [ Google Scholar ]
• Yang, X.; Zhu, Y.; Guo, Y.; Zhou, D. An image super-resolution network based on multi-scale convolution fusion. Vis. Comput. 2022 , 38 , 4307–4317. [ Google Scholar ] [ CrossRef ]
• Hu, H.; Chen, Y.; Xu, J.; Borse, S.; Cai, H.; Porikli, F.; Wang, X. Learning implicit feature alignment function for semantic segmentation. In Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23–27 October 2022; Springer Nature: Cham, Switzerland, 2022; pp. 487–505. [ Google Scholar ]
• Ren, S.; Zhao, N.; Wen, Q.; Han, G.; He, S. Unifying Global-Local Representations in Salient Object Detection with Transformers. IEEE Trans. Emerg. Top. Comput. Intell. 2024 , 8 , 2870–2879. [ Google Scholar ] [ CrossRef ]
• Soille, P.; Vogt, P. Morphological segmentation of binary patterns. Pattern Recognit. Lett. 2009 , 30 , 456–459. [ Google Scholar ] [ CrossRef ]
• Vogt, P.; Ritters, K. GuidosToolbox: Universal digital image object analysis. Eur. J. Remote Sens. 2017 , 50 , 352–361. [ Google Scholar ] [ CrossRef ]
• Xiong, C.; Wu, Z.; Zeng, Z.Y.; Gong, J.Z.; Li, J.T. Spatiotemporal evolution of forest landscape pattern in Guangdong-HongKong-Macao Greater Bay Area based on “Spatial Morphology-Fragmentation-Aggregation”. Acta Ecol. Sin. 2023 , 43 , 3032–3044. [ Google Scholar ]
• Pan, Y.; Jiao, S.; Hu, J.; Guo, Q.; Yang, Y. An Ecological Resilience Assessment of a Resource-Based City Based on Morphological Spatial Pattern Analysis. Sustainability 2024 , 16 , 6476. [ Google Scholar ] [ CrossRef ]
• Zhu, H.; Zhang, B.; Song, W.; Dai, J.; Lan, X.; Chang, X. Power-Weighted Prediction of Photovoltaic Power Generation in the Context of Structural Equation Modeling. Sustainability 2023 , 15 , 10808. [ Google Scholar ] [ CrossRef ]
• Hu, L.; Xu, N.; Liang, J.; Li, Z.; Chen, L.; Zhao, F. Advancing the Mapping of Mangrove Forests at National-Scale Using Sentinel-1 and Sentinel-2 Time-Series Data with Google Earth Engine: A Case Study in China. Remote Sens. 2020 , 12 , 3120. [ Google Scholar ] [ CrossRef ]
• Yang, L.; Driscol, J.; Sarigai, S.; Wu, Q.; Chen, H.; Lippitt, C.D. Google Earth Engine and Artificial Intelligence (AI): A Comprehensive Review. Remote Sens. 2022 , 14 , 3253. [ Google Scholar ] [ CrossRef ]
• Li, L.; Zhu, L.; Xu, N.; Liang, Y.; Zhang, Z.; Liu, J.; Li, X. Climate Change and Diurnal Warming: Impacts on the Growth of Different Vegetation Types in the North–South Transition Zone of China. Land 2022 , 12 , 13. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Wang, F.; Yang, F.; Wang, Z. A Study on the Evolution of Forest Landscape Patterns in the Fuxin Region of China Combining SC-UNet and Spatial Pattern Perspectives. Sustainability 2024 , 16 , 7067. https://doi.org/10.3390/su16167067

Wang F, Yang F, Wang Z. A Study on the Evolution of Forest Landscape Patterns in the Fuxin Region of China Combining SC-UNet and Spatial Pattern Perspectives. Sustainability . 2024; 16(16):7067. https://doi.org/10.3390/su16167067

Wang, Feiyue, Fan Yang, and Zixue Wang. 2024. "A Study on the Evolution of Forest Landscape Patterns in the Fuxin Region of China Combining SC-UNet and Spatial Pattern Perspectives" Sustainability 16, no. 16: 7067. https://doi.org/10.3390/su16167067

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Newsletters
• Memberships
• Enter your keyword and hit 'Enter'

Sign Into Digital Commerce 360

Forgot your password?

Digital Commerce 360 | Retail

Us ecommerce sales reached $1.119 trillion in 2023, u.s. ecommerce represented 22.0% of total retail sales, according to digital commerce 360 analysis of u.s. department of commerce data..

In 2023, U.S. ecommerce represented 22.0% of total retail sales, according to Digital Commerce 360 analysis of U.S. Department of Commerce data . That compares with 21.2% penetration in 2022.

2023’s 22.0% marked the largest U.S. ecommerce sales penetration to date, according to Digital Commerce 360 analysis of U.S. Department of Commerce data. The department’s ecommerce data goes back to the year 2000.

At the same time, U.S. ecommerce grew 7.6% in 2023 and total sales grew 3.8%. That’s a sharp contrast from the pandemic-induced U.S. ecommerce boom, which led online sales to grow at a rate of 42.8% over 2019, whereas total retail sales in 2020 grew 7.8%.

Editor’s note: An earlier version of this chart has been updated to reflect 22.0% penetration for U.S. ecommerce in 2023.

Prevent a Holiday Hangover with Year-Long Performance on CTV

How much did US ecommerce sales grow?

U.S. ecommerce sales grew to about $1.119 trillion in 2023 from $1.040 trillion in 2022 (7.6% growth). Meanwhile, total retail sales grew to about $5.088 trillion in 2023 from about $4.904 trillion in 2022 (3.8%).

U.S. ecommerce has also grown every quarter going back to Q2 2009, when it decreased 3.9% over Q2 2008. Similarly, total retail sales in the U.S. have grown every quarter going back to 2009, according to a Digital Commerce 360 analysis of commerce department data. Total retail sales decreased every quarter that year, as well as in Q4 2008, a result of the Great Recession .

Outside of the Great Recession, total U.S. retail sales have not declined going back at least through 1993, the extent to which Digital Commerce 360 analysis is available.

“Ecommerce growth continued to slow this year amid an overall slower economy, but it accounted for nearly half the total retail growth for the country,” said James Risley, research data manager and senior analyst at Digital Commerce 360. “That’s a return to pre-pandemic levels of contribution compared to a much smaller contribution in 2021 and 2022. Overall, the ecommerce picture is returning to our pre-COVID understanding of retail.”

2024 State of American Ecommerce Report

How is ecommerce penetration calculated?  

U.S. ecommerce sales accounted for 15.4% of total sales in 2023, and 14.7% of total sales in 2022, according to the Commerce Department.

Digital Commerce 360 studies non-seasonally adjusted commerce department data and excludes spending in segments that don’t typically sell online. These segments include:

• Restaurants
• Automobile dealers
• Gas stations
• Fuel dealers

U.S. ecommerce penetration reflects the share of dollars consumers could potentially spend online.

The commerce department defines ecommerce sales as the sales of goods and services where an order is placed by the buyer or price and terms of sales are negotiated over:

• Electronic Data Interchange (EDI) network
• Electronic mail
• Other online system

Payment may or may not be made online. The Commerce Department publishes estimates it adjusts for seasonal variation and holiday and trading-day differences, but not for price changes.

Percentage changes may not align exactly with dollar figures due to rounding. Here’s last year’s update .

Ecommerce sales reach Q1 record share of total sales

Do you rank in our databases? 

Submit your data  and we’ll see where you fit in our next ranking update.

Stay on top of the latest developments in the ecommerce industry. Sign up for a complimentary subscription to  Digital Commerce 360 Retail News . Follow us on  LinkedIn ,  Twitter  and  Facebook . Be the first to know when Digital Commerce 360 publishes news content.

More on This

In This Article

• Retail & Online Retail
• U.S. Ecommerce
• Video included

Related Stories

Holiday online sales grow domestically, globally in 2023

Amazon operating income growth outpaces sales in Q2 2024

Ecommerce earnings recap: What you missed from Allbirds, Hims & Hers and more

Salesforce revenue grows in fiscal Q1 behind AI push

B2B ecommerce market: US sales top $2 trillion in 2023

• About Digital Commerce 360
• Our Products & Solutions
• Free Subscriptions
• Our Research SHOP
• News & Analysis
• Retail Ecommerce News
• B2B Ecommerce News
• Digital Commerce 360 Blog
• Free Industry Reports
• Charts & Infographics
• Vendor Directory
• Return Policy
• Agreement Terms & Conditions
• Privacy Policy
• Terms of Use
• Accessibility Statement
• Website Membership Login
• Database Login
• Connect with Us
• Advertise With Us

Copyright © 2024 Digital Commerce 360 | Vertical Web Media LLC