Automatic segmentation of skin cells in multiphoton data using multi-stage merging

Prinke P., Haueisen J., Klee S., Rizqie M.Q., Supriyanto E., König K., Breunig H.G., Piątek Ł.

Institute for Biomedical Engineering and Informatics, Technische Universität Ilmenau, Ilmenau, 98693, Germany; Division Biostatistics and Data Science, Department of General Health Studies, Karl Landsteiner University of Health Sciences, Dr. Karl-Dorrek-Straße 30, Krems, 3500, Austria; Informatics Engineering Program, Universitas Sriwijaya, Palembang, South Sumatera, Indonesia; IJN-UTM Cardiovascular Engineering Centre, Universiti Teknologi Malaysia, Johor Bahru, Johor, Malaysia; Department of Biophotonics and Laser Technology, Saarland University, Campus A5.1, Saarbrücken, 66123, Germany; JenLab GmbH, Johann-Hittorf-Straße 8, Berlin, 12489, Germany; Department of Artificial Intelligence, University of Information Technology and Management, H. Sucharskiego 2 Str, Rzeszów, 35-225, Poland


We propose a novel automatic segmentation algorithm that separates the components of human skin cells from the rest of the tissue in fluorescence data of three-dimensional scans using non-invasive multiphoton tomography. The algorithm encompasses a multi-stage merging on preprocessed superpixel images to ensure independence from a single empirical global threshold. This leads to a high robustness of the segmentation considering the depth-dependent data characteristics, which include variable contrasts and cell sizes. The subsequent classification of cell cytoplasm and nuclei are based on a cell model described by a set of four features. Two novel features, a relationship between outer cell and inner nucleus (OCIN) and a stability index, were derived. The OCIN feature describes the topology of the model, while the stability index indicates segment quality in the multi-stage merging process. These two new features, combined with the local gradient magnitude and compactness, are used for the model-based fuzzy evaluation of the cell segments. We exemplify our approach on an image stack with 200 × 200 × 100 μm3, including the skin layers of the stratum spinosum and the stratum basale of a healthy volunteer. Our image processing pipeline contributes to the fully automated classification of human skin cells in multiphoton data and provides a basis for the detection of skin cancer using non-invasive optical biopsy. © 2021, The Author(s).


Scientific Reports

Publisher: Nature Research

Volume 11, Issue 1, Art No 14534, Page – , Page Count

Journal Link:

doi: 10.1038/s41598-021-93682-y

Issn: 20452322

Type: All Open Access, Gold, Green


Tejera-Vaquerizo, A., Chronology of metastasis in cutaneous melanoma: growth rate model (2012) J. Invest. Dermatol., 132, pp. 1215-1221. , COI: 1:CAS:528:DC%2BC38XivFejtA%3D%3D, PID: 22217744; (1975) Melanoma of the skin. SEER Cancer Statistics Review; König, K., (2018) Multiphoton Microscopy and Fluorescence Lifetime Imaging, Applications in Biology and Medicine, , De Gruyter; Denk, W., Strickler, J.H., Webb, W.W., Two-photon laser scanning fluorescence microscopy (1990) Science, 248, p. 73. , COI: 1:CAS:528:DyaK3cXktVaktb8%3D, PID: 2321027; Freund, I., Deutsch, M., Second-harmonic microscopy of biological tissue (1986) Opt. Lett., 11, pp. 94-96. , COI: 1:STN:280:DC%2BD1MngtlOqsA%3D%3D, PID: 19730544; Lakowicz, J.R., Berndt, K.W., Lifetime-selective fluorescence imaging using an rf phase-sensitive camera (1991) Rev. Sci. Instrum., 62, pp. 1727-1734. , COI: 1:CAS:528:DyaK3MXks1Olsbw%3D; Peters, S., Griebsch, M., Klemm, M., Haueisen, J., Hammer, M., Hydrogen peroxide modulates energy metabolism and oxidative stress in cultures of permanent human Müller cells MIO-M1 (2016) J. Biophotonics, 10, pp. 1180-1188. , PID: 27896951; Klemm, M., FLIMX: a software package to determine and analyze the fluorescence lifetime in time-resolved fluorescence data from the human eye (2015) PLoS ONE, 10. , PID: 26192624; König, K., Riemann, I., High-resolution multiphoton tomography of human skin with subcellular spatial resolution and picosecond time resolution (2003) J. Biomed. Opt., 8, pp. 432-439. , PID: 12880349; Seidenari, S., Multiphoton laser microscopy and fluorescence lifetime imaging for the evaluation of the skin (2012) Dermatol. Res. Pract.; König, K., Review: clinical in vivo multiphoton FLIM tomography (2020) Methods Appl. Fluoresc., 8, p. 34002; (2003) Optical Biopsy, ,; Wang, T.D., van Dam, J., Optical biopsy: a new frontier in endoscopic detection and diagnosis (2004) Clin. Gastroenterol. Hepatol., 2, pp. 744-753. , PID: 15354274; König, K., Translation of two-photon microscopy to the clinic: multimodal multiphoton CARS tomography of in vivo human skin (2020) J. Biomed. Opt., 25, pp. 1-12. , PID: 32006419; Walsh, A., Skala, M., An automated image processing routine for segmentation of cell cytoplasms in high-resolution autofluorescence images (2014) In Progress in Biomedical Optics and Imaging—Proceedings of SPIE; Decencière, E., Koudoro, S., Brizion, S., Pena, A.-M., Baldeweck, T., A 3d automatic segmentation method based on mathematical morphology for multiphoton images of melanocyte-keratinocyte coculture skin model (2015) Focus on Microscopy (Göttingen; Decencière, E., Automatic 3D segmentation of multiphoton images: a key step for the quantification of human skin (2013) Skin Res. Technol. Off. J. Int. Soc. Bioeng. Skin (ISBS) Int. Soc. Digital Imaging Skin Int. Soc. Skin Imaging (ISSI), 19, pp. 115-124; Wu, W., A novel multiphoton microscopy images segmentation method based on superpixel and watershed (2017) J. Biophotonics, 10, pp. 532-541. , COI: 1:CAS:528:DC%2BC2sXlsleru70%3D, PID: 27090206; Chen, G., Lui, H., Zeng, H., Image segmentation for integrated multiphoton microscopy and reflectance confocal microscopy imaging of human skin in vivo (2015) Quant. Imaging Med. Surg., 5, pp. 17-22. , PID: 25694949; Bates, R., Segmentation of vasculature from fluorescently labeled endothelial cells in multi-photon microscopy images (2019) IEEE Trans. Med. Imaging, 38, pp. 1-10. , PID: 28796613; Machairas, V., Baldeweck, T., Walter, T., Decenciere, E., New general features based on superpixels for image segmentation learning (2016) In IEEE 13Th International Symposium on Biomedical Imaging (ISBI); Hu, D., Automatic segmentation of fluorescence lifetime microscopy images of cells using multiresolution community detection: a first study (2014) J. Microsc., 253, pp. 54-64. , COI: 1:CAS:528:DC%2BC3sXhvFalu77J, PID: 24251410; Arce, S.H., Wu, P.-H., Tseng, Y., Fast and accurate automated cell boundary determination for fluorescence microscopy (2013) Sci. Rep., 3, p. 2266. , PID: 23881180; Arslan, S., Ersahin, T., Cetin-Atalay, R., Gunduz-Demir, C., Attributed relational graphs for cell nucleus segmentation in fluorescence microscopy images (2013) IEEE Trans. Med. Imaging, 32, pp. 1121-1131. , PID: 23549886; Maška, M., Segmentation and shape tracking of whole fluorescent cells based on the Chan-Vese model (2013) IEEE Trans. Med. Imaging, 32, pp. 995-1006. , PID: 23372077; Chan, T.F., Vese, L.A., Active contours without edges (2001) IEEE Trans. Image Process., 10, pp. 266-277. , COI: 1:STN:280:DC%2BD1c%2FptFeltA%3D%3D, PID: 18249617; Tarnawski, W., A robust algorithm for segmenting and tracking clustered cells in time-lapse fluorescent microscopy (2013) IEEE J. Biomed. Health Inform., 17, pp. 862-869. , PID: 25055315; Chen, C., Wang, W., Ozolek, J.A., Rohde, G.K., A flexible and robust approach for segmenting cell nuclei from 2D microscopy images using supervised learning and template matching (2013) Cytom. A J. Int. Soc. Anal. Cytol., 83, pp. 495-507; Dimopoulos, S., Mayer, C.E., Rudolf, F., Stelling, J., Accurate cell segmentation in microscopy images using membrane patterns (2014) Bioinformatics (Oxford, England), 30, pp. 2644-2651. , COI: 1:CAS:528:DC%2BC2cXhslehsrfM; Wienert, S., Detection and segmentation of cell nuclei in virtual microscopy images: a minimum-model approach (2012) Sci. Rep., 2, p. 503. , PID: 22787560; (2013) Automated Nucleus and Cytoplasm Segmentation of Overlapping Cervical Cells, , Springer, Berlin; Chen, Y.-F., Semi-automatic segmentation and classification of Pap smear cells (2014) IEEE J. Biomed. Health Inform., 18, pp. 94-108. , COI: 1:CAS:528:DC%2BC2cXitFKktLjN, PID: 24403407; Dima, A.A., Comparison of segmentation algorithms for fluorescence microscopy images of cells (2011) Cytom. A J. Int. Soc. Anal. Cytol., 79, pp. 545-559; Meijering, E., Cell segmentation: 50 years down the road [life sciences] (2012) IEEE Signal Process. Mag., 29, pp. 140-145; Guimarães, P., Batista, A., Zieger, M., Kaatz, M., Koenig, K., Artificial intelligence in multiphoton tomography: atopic dermatitis diagnosis (2020) Sci. Rep., 10, p. 7968. , PID: 32409755; Masters, B.R., So, P.T., Gratton, E., Multiphoton excitation fluorescence microscopy and spectroscopy of in vivo human skin (1997) Biophys. J., 72, pp. 2405-2412. , COI: 1:CAS:528:DyaK2sXjsVOns7k%3D, PID: 9168018; König, K., Applications of multiphoton tomographs and femtosecond laser nanoprocessing microscopes in drug delivery research (2011) Adv. Drug Delivery Rev., 63, pp. 388-404; König, K., High-resolution multimodal clinical multiphoton tomography of skin (2011) In Progress in Biomedical Optics and Imaging Proceedings of SPIE, 7883; Barel, A.O., Paye, M., Maibach, H.I., Handbook of Cosmetic Science and Technology. 3rd (2009) (Informa Healthcare; Hendriks, F.M., Mechanical Behaviour of Human Skin in Vivo: A Literature Review, , 07/2001; Egawa, M., Hirao, T., Takahashi, M., In vivo estimation of stratum corneum thickness from water concentration profiles obtained with Raman spectroscopy (2007) Acta Derm. Venereol., 87, pp. 4-8. , PID: 17225007; Dice, L.R., Measures of the amount of ecologic association between species (1945) Ecology, 26, pp. 297-302; Serra, J., (1983) Image Analysis and Mathematical Morphology, , Academic Press, Inc; Pizer, S.M., Adaptive histogram equalization and its variations (1987) Comput. Vision Graph. Image Process., 39, pp. 355-368; Perona, P., Malik, J., Scale-space and edge detection using anisotropic diffusion (1990) IEEE Trans. Pattern Anal. Mach. Intell., 12, pp. 629-639; Meyer, F., Beucher, S., Morphological segmentation (1990) J. Vis. Commun. Image Represent., 1, pp. 21-46; Ren, X., Malik, J., Learning a classification model for segmentation (2003) In Proceedings Ninth IEEE International Conference on Computer Vision; Haralick, R.M., Shanmugam, K., Dinstein, I.H., Textural features for image classification (1973) IEEE Trans. Syst. Man Cybern., 3, pp. 610-621

Indexed by Scopus

Leave a Comment