Home


Research


Publication


Download


Teaching


Links

Big Image-Omics Data Analytics for Clinical Outcome Prediction

Recent technological innovations are enabling scientists to capture complex image-omics data from different views. However, the major computational challenges are due to the unprecedented scale and complexity of heterogeneous image-omics data analytics. To solve the key and challenging problems in mining such comprehensive heterogeneous image-omics data, this project proposes to develop novel large scale learning tools and explore ways to integrate features from multiple data sources for clinical outcome prediction. It will greatly support the Precision Medicine Initiative, which has become a national goal and was unveiled by the U.S. government as a research effort designed to enable physicians to select individualized treatments.

Recent studies demonstrated the feasibility and advantage of using digital pathological image analysis for objective and unbiased clinical prognosis. However, there is a lack of comprehensive pathological image analysis for cancer data due to the complexity and heterogeneity of the disease. With the advance of technology, tumor tissue histology slide scanning is becoming a routine clinical procedure, which produces massive digital pathological images that capture histological details in high resolution. In this study, we will develop novel and powerful computational approaches to analyze pathological images. We will also develop algorithms to integrate features from pathological images with clinical and molecular profiling data to predict the clinical outcomes of cancer patients.

Web Demo

Publication

  1. Zheng Xu, Junzhou Huang, "Efficient Lung Cancer Cell Detection with Deep Convolution Neural Network", 1st International Workshop on Patch-based Techniques in Medical Imaging, PMI'15, Munich, Germany, October 2015.
  2. Hao Pan, Zheng Xu, Junzhou Huang, "An Effective Approach for Robust Lung Cancer Cell Detection", 1st International Workshop on Patch-based Techniques in Medical Imaging, PMI'15, Munich, Germany, October 2015.
  3. Ruoyu Li, Junzhou Huang, "Fast Regions-of-Interest Detection in Whole Slide Histopathology Images", 1st International Workshop on Patch-based Techniques in Medical Imaging, PMI'15, Munich, Germany, October 2015.
  4. Jiawen Yao, Dheeraj Ganti, Xin Luo, Guanghua Xiao, Yang Xie, Shirley Yan and Junzhou Huang, "Computer-assisted Diagnosis of Lung Cancer Using Quantitative Topology Features", 6th International Workshop on Machine Learning in Medical Imaging, MLMI'15, Munich, Germany, October 2015.
  5. Menglin Jiang, Shaoting Zhang, Junzhou Huang, Lin Yang, Dimitris Metaxas, "Joint Kernel-Based Supervised Hashing for Scalable Histopathological Image Analysis", In Proc. of the 18th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'15, Munich, Germany, October 2015. (MICCAI Young Scientist Award, 4 out of 810 submissions)
  6. Xinliang Zhu, Jianwen Yao, Xin Luo, Guanghua Xiao, Yang Xie, Adi Gazdar and Junzhou Huang, "Lung Cancer Survival Prediction from Pathological Images and Genetic Data - An Integration Study", In Proc. of The International Symposium on Biomedical Imaging, ISBI'16, Prague, Czech Republic, April 2016.
  7. Zheng Xu, Junzhou Huang, "Detecting 10,000 Cells in One Second", In Proc. of the 19th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'16, Athens, Greece, October 2016.
  8. Sheng Wang, Jiawen Yao, Zheng Xu, Junzhou Huang, "Subtype Cell Detection with an Accelerated Deep Convolution Neural Network", In Proc. of the 19th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'16, Athens, Greece, October 2016.
  9. Jiawen Yao, Sheng Wang, Xinliang Zhu, Junzhou Huang, "Clinical Imaging Biomarker Discovery for Survival Prediction on Lung Cancer Imaging Genetic Data", In Proc. of the 19th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'16, Athens, Greece, October 2016. (Oral Presentation)
  10. Menglin Jiang, Shaoting Zhang, Junzhou Huang, Lin Yang, Dimitris Metaxas, "Scalable Histopathological Image Analysis via Supervised Hashing with Multiple Features", Medical Image Analysis, Volume 34, pp. 3-12, December 2016.
  11. Xinliang Zhu, Jiawen Yao and Junzhou Huang, "Deep Convolutional Neural Network for Survival Analysis with Pathological Images", In Proc. of IEEE International Conference on Bioinformatics and Biomedicine, BIBM'16, Shenzhen, China, December 2016.
  12. Xinliang Zhu, Jiawen Yao, Guanghua Xiao, Yang Xie, Jaime Rodriguez-Canales, Edwin R. Parra, Carmen Behrens, Ignacio I. Wistuba and Junzhou Huang, "Imaging-Genetic Data Mapping for Clinical Outcome Prediction via Supervised Conditional Gaussian Graphical Model", In Proc. of IEEE International Conference on Bioinformatics and Biomedicine, BIBM'16, Shenzhen, China, December 2016.
  13. Jiawen Yao, Xinliang Zhu, Feiyun Zhu and Junzhou Huang, “Deep Correlational Learning for Survival Prediction from Multi-modality Data”, In Proc. of the 19th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI’17, Quebec City, Quebec, Canada, September 2017. (Oral Presentation)
  14. Xin Luo, Faliu Yi, Junzhou Huang, Lin Yang, Yang Xie and Guanghua Xiao, “Automatic Extraction of Cell Nuclei from H&E-stained Histopathological Images”, Journal of Medical Imaging, Volume 4, pp. 4-12, June 2017.
  15. Xin Luo, Xiao Zang, Lin Yang, Junzhou Huang, Faming Liang, Jaime Rodriguez Canales, Ignacio I. Wistuba, Adi Gazdar, Yang Xie, Guanghua Xia, "Comprehensive Computational Pathological Image Analysis Predicts Lung Cancer Prognosis", Journal of Thoracic Oncology, Volume 12, pp.501-509, March 2017.
  16. Xinliang Zhu, Jiawen Yao, Feiyun Zhu and Junzhou Huang, “WSISA: Making Survival Prediction from Whole Slide Pathology Images”, In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, CVPR’17, Honolulu, Hawaii, USA, July 2017.
  17. Ruoyu Li, Sheng Wang, Feiyun Zhu and Junzhou Huang, "Adaptive Graph Convolutional Neural Networks", In Proc. of The Thirty-Second AAAI Conference on Artificial Intelligence, AAAI'18, New Orleans, USA, February 2018. (Oral Presentation) [CODE]
  18. Ruoyu Li, Jiawen Yao, Xinliang Zhu, Yeqing Li and Junzhou Huang, "Graph CNN for Survival Analysis on Whole Slide Pathological Images", In Proc. of the 20th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'18, Granada, Spain, September 2018. [CODE]
  19. Chao Li, Xinggang Wang, Wenyu Liu, Longin Jan Lateckib, Bo Wang and Junzhou Huang, "Weakly Supervised Mitosis Detection in Breast Histopathology Images Using Concentric Loss", Medical Image Analysis, Volume 53, pp.165-178, April 2019.
  20. Chaoqi Chen, Weiping Xie, Tingyang Xu, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang and Junzhou Huang, “Progressive Feature Alignment for Unsupervised Domain Adaptation", In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, CVPR'19, Long Beach, CA, USA, June 2019.
  21. Yifan Zhang, Hanbo Chen, Ying Wei, Peilin Zhao, Jiezhang Cao, Mingkui Tan, Qingyao Wu, Xinjuan Fan, Xiaoying Lou, Hailing Liu, Jinlong Hou, Xiao Han, Jianhua Yao and Junzhou Huang, "From Whole Slide Imaging to Microscopy: Deep Microscopy Adaptation Network for Histopathology Cancer Image Classification", In Proc. of the 21st Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'19, Shenzhen, China, October 2019.
  22. Hanbo Chen, Xiao Han, Xinjuan Fan, Xiaoying Lou, Hailing Liu, Junzhou Huang and Jianhua Yao, "Rectified Cross-Entropy and Upper Transition Loss for Weakly Supervised Whole Slide Image Classifier", In Proc. of the 21st Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'19, Shenzhen, China, October 2019.
  23. Jiawen Yao, Xinliang Zhu and Junzhou Huang, "Deep Multi-Instance Learning for survival prediction from Whole Slide Images", In Proc. of the 21st Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'19, Shenzhen, China, October 2019. [CODE]
  24. Ashwin Raju, Jiawen Yao, Mohammad Minhazul Haq, Jitendra Jonnagaddala and Junzhou Huang, "Graph Attention Multi-instance Learning for Accurate Colorectal Cancer Staging", In Proc. of the 22nd Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'20, Lima, Peru, October 2020.
  25. Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins and Junzhou Huang, "Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Networks", Medical Image Analysis, Volume 65, October 2020. [CODE]
  26. Siqi Yang, Jun Zhang, Junzhou Huang, Brian Lovell and Xiao Han, “Minimizing Labeling Cost for Nuclei Instance Segmentation and Classificationwith Cross domain Images and Weak Labels”, In Proc. of the Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI’21, Vancouver, Canada, February 2021.
  27. Zhen Chen, Jun Zhang, Shuanlong Che, Junzhou Huang, Xiao Han, Yixuan Yuan, “Diagnose Like A Pathologist: Weakly-Supervised Pathologist-Tree Network for Slide-Level Immunohistochemical Scoring”, In Proc. of the Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI’21, Vancouver, Canada, February 2021.
  28. Lijing Cai, Kezhou Yan, Hong Bu, Meng Yue, Pei Dong, Xinran Wang, Lina Li, Kuan Tian, Haocheng Shen, Jun Zhang, Jiuyan Shang, Shuyao Niu, Dandan Han, Chen Ren, Junzhou Huang, Xiao Han, Jianhua Yao, Yueping Liu, ``Improving Ki-67 Assessment Concordance with AI-Empowered Microscope: A Multi-institutional Ring Study", Histopathology, April 2021.
  29. Yueping Liu, Xinran Wang, Liang Wang, Hong Bu, Ningning Zhang, Meng Yue, Zhanli Jia, Lijing Cai, Jiankun He, Yanan Wang, Xin Xu, Shengshui Li, Kaiwen Xiao, Kezhou Yan, Kuan Tian, Xiao Han, Junzhou Huang and Jianhua Yao, "How Can Artificial Intelligence Models Assist PD-L1 Expression Scoring in Breast Cancer: Results of Multi-institutional Ring Studies", NPJ Breast Cancer 7, 61, Volume 7, Issue 1, May 2021.
  30. Meng Yue, Jun Zhang, Xinran Wang, Kezhou Yan, Lijing Cai, Kuan Tian, Shuyao Niu, Xiao Han, Yongqiang Yu, Junzhou Huang, Dandan Han, Jianhua Yao, Yueping Liu, "Can AI-assisted Microscope Facilitate Breast HER2 Interpretation? A Multi-institutional Ring Study”, Virchows Archiv, 479, pp. 443-449, July 2021.
  31. Hang Li, Fan Yang, Yu Zhao, Xiaohan Xing, Jun Zhang, Mingxuan Gao, Junzhou Huang, Liansheng Wang and Jianhua Yao, "DT-MIL: Deformable Transformer for Multi-instance Learning on Histopathological Image", In Proc. of the 24th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'21, Strasbourg, France, September 2021.
  32. Jiangpeng Yan, Hanbo Chen, Kang Wang, Yan Ji, Yuyao Zhu, Jingjing Li, Dong Xie, Zhe Xu, Junzhou Huang, Shuqun Cheng, Xiu Li and Jianhua Yao, "Hierarchical Attention Guided Framework for Multi-resolution Collaborative Whole Slide Image Segmentation", In Proc. of the 24th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'21, Strasbourg, France, September 2021.
  33. Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang and Xiao Han, "TransPath: Transformer-based Self-supervised Learning for Histopathological Image Classification", In Proc. of the 24th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'21, Strasbourg, France, September 2021.
  34. Hanbo Chen, Kang Wang, Yuyao Zhu, Jiangpeng Yan, Yan Ji, Jingjing Li, Dong Xie, Junzhou Huang, Shuqun Cheng and Jianhua Yao, "From Pixel to Whole Slide: Automatic Detection of Microvascular Invasion in Hepatocellular Carcinoma on Histopathological Image via Cascaded Networks", In Proc. of the 24th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'21, Strasbourg, France, September 2021.
  35. Hang Li, Fan Yang, Xiaohan Xing, Yu Zhao, Jun Zhang, Yueping Liu, Mengxue Han, Junzhou Huang, Liansheng Wang and Jianhua Yao, "Multi-modal Multi-instance Learning using Weakly Correlated Histopathological Images and Tabular Clinical Information", In Proc. of the 24th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'21, Strasbourg, France, September 2021.
  36. Hailing Liu, Yu Zhao, Fan Yang, Xiaoying Lou, Feng Wu, Hang Li, Xiaohan Xing, Tingying Peng, Bjoern Menze, Junzhou Huang, Shujun Zhang, Anjia Han, Jianhua Yao and Xinjuan Fan, "Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer with Deep Learning", BME Frontiers, Volume 2022, pp. 1-12, March 2022.
  37. Xiaohan Xing, Fan Yang, Hang Li, Jun Zhang, Yu Zhao, Mingxuan Gao, Junzhou Huang and Jianhua Yao, "Multi-Level Attention Graph Neural Network Based on Co-expression Gene Modules for Disease Diagnosis and Prognosis", Bioinformatics, Volume 38, Issue 8, pp. 2178-2186, April 2022.
  38. Xiaoying Lou, Niyun Zhou, Lili Feng, Zhenhui Li, Yuqi Fang, Xinjuan Fan, Yihong Ling, Hailing Liu, Xuan Zou, Jing Wang, Junzhou Huang, Jingping Yun, Jianhua Yao and Yan Huang, "Deep Learning Model for Predicting the Pathological Complete Response to Neoadjuvant Chemoradiotherapy of Locally Advanced Rectal Cancer", Frontiers in Oncology, June 2022.
  39. Yonghang Guan, Jun Zhang, Kuan Tian, Sen Yang, Pei Dong, Jinxi Xiang, Wei Yang, Junzhou Huang, Yuyao Zhang and Xiao Han, "Node-aligned Graph Convolutional Network for Whole-slide Image Representation and Classification", In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, CVPR'22, New Orleans, Louisiana, USA, June 2022.
  40. Zongbo Han, Fan Yang, Junzhou Huang, Changqing Zhang and Jianhua Yao, "Multimodal Dynamics: Dynamical Fusion for Trustworthy Multimodal Classification", In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, CVPR'22, New Orleans, Louisiana, USA, June 2022.
  41. Chunyuan Li, Xinliang Zhu, Jiawen Yao and Junzhou Huang, "Hierarchical Transformer for Survival Prediction Using Multi-modality Whole Slide Images and Genomics", In Proc. of the 26th International Conference on Pattern Recognition, ICPR’22, Montral Qubec, Canada, August 2022.
  42. Piumi Sandarenu, Ewan KA Millar, Yang Song, Lois H Browne, Julia Beretov, Jodi Lynch, Peter Graham, Jitendra Jonnagaddala, Nick Hawkins, Junzhou Huang and Erik Meijering, "Survival Prediction in Triple Negative Breast Cancer Using Multiple Instance Learning of Histopathological Images", Scientific Reports, Volume 12, August 2022.
  43. Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu and Jianhua Yao, "scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data", Nature Machine Intelligence, Volume 4, pp. 852- 866, September 2022.
  44. Yu Zhao, Zhenyu Lin, Kai Sun, Yidan Zhang, Junzhou Huang, Liansheng Wang and Jianhua Yao, "SETMIL: Spatial Encoding Transformer-based Multiple Instance Learning for Pathological Image Analysis", In Proc. of the 25th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI’22, Singapore, September 2022.
  45. Mohammad Minhazul Haq and Junzhou Huang, "Self-Supervised Pre-Training for Nuclei Segmentation", In Proc. of the 25th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI’22, Singapore, September 2022.
  46. Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang and Xiao Han, “Transformer-based Unsupervised Contrastive Learning for Histopathological Image Classification”, Medical Image Analysis, Volume 81, October 2022.
  47. Jiawei Yang, Hanbo Chen, Yuan Liang, Junzhou Huang, Lei He and Jianhua Yao, "ConCL: Concept Contrastive Learning for Dense Prediction Pre-training in Pathology Images", In Proc. of the 17th European Conference on Computer Vision, ECCV'22, Tel Aviv, Israel, October 2022.
  48. Xiyue Wang, Jinxi Xiang, Jun Zhang, Sen Yang, Zhongyi Yang, Ming-Hui Wang, Jing Zhang, Yang Wei, Junzhou Huang, Xiao Han, "SCL-WC: Cross-Slide Contrastive Learning for Weakly-Supervised Whole-Slide Image Classification", In Proc. of the 36th Annual Conference on Neural Information Processing Systems, NeurIPS'22, New Orleans, Louisiana, USA, December 2022.
  49. Xiyue Wang, Yuexi Du, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang and Xiao Han, "RetCCL: Clustering-guided Contrastive Learning for Whole-slide Image Retrieval", Medical Image Analysis, Volume 83, January 2023.
  50. Wenhua Zhang, Jun Zhang, Xiyue Wang, Sen Yang, Junzhou Huang, Wei Yang, Wenping Wang and Xiao Han, "Merging Nucleus Datasets by Correlation-based Cross-Training”, Medical Image Analysis, Volume 84, February 2023.
  51. Xiyue Wang, Jun Zhang, Sen Yang, Jingxi Xiang, Feng Luo, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang and Xiao Han, "A Generalizable and Robust Deep Learning Algorithm for Mitosis Detection in Multicenter Breast Histopathological Images", Medical Image Analysis, Volume 84, February 2023.
  52. Mohammad Minhazul Haq, Hehuan Ma and Junzhou Huang, "NuSegDA: Domain Adaptation for Nuclei Segmentation", Frontiers in Big Data-Medicine and Public Health, Volume 6, February 2023.
  53. Qin Ren, Yu Zhao, Bing He, Bingzhe Wu, Sijie Mai, Fan Xu, Yueshan Huang, Yonghong He, Junzhou Huang and Jianhua Yao, "IIB-MIL: Integrated Instance-level and Bag-level Multiple Instances Learning with Label Disambiguation for Pathological Image Analysis", In Proc. of the 26th Annual International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI'23, Vancouver, Canada, October 2023.

 
Figure 1. Image-Omics Data: Pathological Image, Gene Mutation, CNV, mRNA Expression, Protein Expression

 

Efficient Cell Detection in the Whole Slide Pathological Images
Figure 2. Cell detection in a whole slide image (13483 x 17943); Time: ~200 seconds; Desktop Computer: Nvidia Tesla K40c, Regular 5400 RPM HDD