[en] Multitask learning (MTL) aims to learn multiple tasks simultaneously while exploiting their mutual relationships. By using shared resources to simultaneously calculate multiple outputs, this learning paradigm has the potential to have lower memory requirements and inference times compared to the traditional approach of using separate methods for each task. Previous work in MTL has mainly focused on fully supervised methods, as task relationships (TRs) can not only be leveraged to lower the level of data dependency of those methods but also improve the performance. However, MTL introduces a set of challenges due to a complex optimization scheme and a higher labeling requirement. This article focuses on how MTL could be utilized under different partial supervision settings to address these challenges. First, this article analyses how MTL traditionally uses different parameter sharing techniques to transfer knowledge in between tasks. Second, it presents different challenges arising from such a multiobjective optimization (MOO) scheme. Third, it introduces how task groupings (TGs) can be achieved by analyzing TRs. Fourth, it focuses on how partially supervised methods applied to MTL can tackle the aforementioned challenges. Lastly, this article presents the available datasets, tools, and benchmarking results of such methods. The reviewed articles, categorized following this work, are available at https://github.com/Klodivio355/MTL-CV-Review .
Disciplines :
Sciences informatiques
Auteur, co-auteur :
Fontana, Maxime ; Department of Informatics, King’,s College London, London, U.K.
SPRATLING, Michael ; University of Luxembourg ; Department of Informatics, King’,s College London, London, U.K.
Shi, MJ ; College of Electronic and Information Engineering and Shanghai Institute of Intelligent Science and Technology, Tongji University, Shanghai, China
Co-auteurs externes :
yes
Langue du document :
Anglais
Titre :
When Multitask Learning Meets Partial Supervision: A Computer Vision Review
Date de publication/diffusion :
2024
Titre du périodique :
Proceedings of the IEEE
ISSN :
0018-9219
eISSN :
1558-2256
Maison d'édition :
Institute of Electrical and Electronics Engineers (IEEE)
A. Bruno, D. Moroni, and M. Martinelli, "Efficient adaptive ensembling for image classification, " 2022, arXiv:2206.07394.
K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition, " in Proc. Int. Conf. Learn. Represent., 2015, pp. 1-14.
G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 4700-4708.
M. Tan and Q. V. Le, "EfficientNet: Rethinking model scaling for convolutional neural networks, " in Proc. Int. Conf. Mach. Learn., 2020, pp. 6105-6114.
W. Wang et al., "InternImage: Exploring large-scale vision foundation models with deformable convolutions, " 2022, arXiv:2211.05778.
R. Girshick, "Fast R-CNN, " in Proc. IEEE Int. Conf. Comput. Vis., Oct. 2015, pp. 1440-1448.
O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional networks for biomedical image segmentation, " in Proc. 18th Int. Conf., 2015, pp. 234-241.
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 770-778.
Y. Shinya, "USB: Universal-scale object detection benchmark, " 2021, arXiv:2103.14027.
J. Redmon and A. Farhadi, "YOLOv3: An incremental improvement, " 2018, arXiv:1804.02767.
R. Caruana, "Multitask learning, " Mach. Learn., vol. 28, pp. 41-75, Jul. 1997.
M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks, " 2013, arXiv:1311.2901.
I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, "Cross-stitch networks for multi-task learning, " 2016, arXiv:1604.03539.
R. Ando and T. Zhang, "A framework for learning predictive structures from multiple tasks and unlabeled data, " J. Mach. Learn. Res., vol. 6, pp. 1817-1853, Nov. 2005.
J. Bingel and A. Sogaard, "Identifying beneficial task relations for multi-task learning in deep neural networks, " in Proc. 15th Conf. Eur. Chapter Assoc. Comput. Linguistics, Jan. 2017, pp. 164-169.
W.-H. Li, X. Liu, and H. Bilen, "Learning multiple dense prediction tasks from partially annotated data, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 18857-18867.
Y. Wang, Y.-H. Tsai, W.-C. Hung, W. Ding, S. Liu, and M.-H. Yang, "Semi-supervised multi-task learning for semantics and depth, " 2021, arXiv:2110.07197.
Q. Liu, X. Liao, and L. Carin, "Semi-supervised multitask learning, " in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, vol. 20. Curran Associates, 2007, pp. 1-8. [Online]. Available: https://proceedings.neurips.cc/paper/2007/file/a34bacf839b923770b2c360eefa26748-Paper.pdf
N. Khosravan and U. Bagci, "Semi-supervised multi-task learning for lung cancer diagnosis, " in Proc. 40th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. (EMBC), Jul. 2018, pp. 710-713.
S. Chowdhuri, T. Pankaj, and K. Zipser, "MultiNet: Multi-modal multi-task learning for autonomous driving, " 2017, arXiv:1709.05581.
K. Ishihara, A. Kanervisto, J. Miura, and V. Hautamaki, "Multi-task learning with attention for end-to-end autonomous driving, " 2021, arXiv:2104.10753.
X. Liang, Y. Wu, J. Han, H. Xu, C. Xu, and X. Liang, "Effective adaptation in multi-task co-training for unified autonomous driving, " 2022, arXiv:2209.08953.
A. Karpathy, "Multi-task learning in the wilderness, " in Proc. ICML, 2019. [Online]. Available: https://slideslive.com/38917690/multitask-learning-in-the-wilderness
M. Islam, V. S. Vibashan, and H. Ren, "AP-MTL: Attention pruned multi-task learning model for real-time instrument detection and segmentation in robot-assisted surgery, " in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), May 2020, pp. 8433-8439, doi: 10.1109/ICRA40945.2020.9196905.
M. Islam, V. S. Vibashan, C. M. Lim, and H. Ren, "ST-MTL: Spatio-temporal multitask learning model to predict scanpath while tracking instruments in robotic surgery, " 2021, arXiv:2112.08189.
Z. Ming, J. Xia, M. M. Luqman, J.-C. Burie, and K. Zhao, "Dynamic multi-task learning for face recognition with facial expression, " 2019, arXiv:1911.03281.
Q. Zheng, J. Deng, Z. Zhu, Y. Li, and S. Zafeiriou, "Decoupled multi-task learning with cyclical self-regulation for face parsing, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 4146-4155.
C. Zhang, X. Hu, Y. Xie, M. Gong, and B. Yu, "A privacy-preserving multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition, " Frontiers Neurorobotics, vol. 13, p. 112, Jan. 2020. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fnbot.2019.00112
Z. Huang, J. Zhang, and H. Shan, "When age-invariant face recognition meets face age synthesis: A multi-task learning framework, " 2021, arXiv:2103.01520.
Z. Xu et al., "Upper gastrointestinal anatomy detection with multi-task convolutional neural networks, " Healthcare Technol. Lett., vol. 6, no. 6, pp. 176-180, Dec. 2019.
Z. Kong et al., "Multi-task classification and segmentation for explicable capsule endoscopy diagnostics, " Frontiers Mol. Biosciences, vol. 8, Aug. 2021, Art. no. 614277.
X. Yu, S. Tang, C. F. Cheang, H. H. Yu, and I. C. Choi, "Multi-task model for esophageal lesion analysis using endoscopic images: Classification with image retrieval and segmentation with attention, " Sensors, vol. 22, no. 1, p. 283, Dec. 2021. [Online]. Available: https://www.mdpi.com/1424-8220/22/1/283
S. M. Kamrul Hasan and C. A. Linte, "A multi-task cross-task learning architecture for ad-hoc uncertainty estimation in 3D cardiac MRI image segmentation, " 2021, arXiv:2109.07702.
X. Liu, P. He, W. Chen, and J. Gao, "Multi-task deep neural networks for natural language understanding, " 2019, arXiv:1901.11504.
J. Pilault, A. E. Hattami, and C. Pal, "Conditionally adaptive multi-task learning: Improving transfer learning in NLP using fewer parameters & less data, " in Proc. Int. Conf. Learn. Represent., 2021, pp. 1-22. [Online]. Available: https://openreview.net/forum?id=de11dbHzAMF
G. Aguilar, S. Maharjan, A. P. Lopez Monroy, and T. Solorio, "A multi-task approach for named entity recognition in social media data, " in Proc. 3rd Workshop Noisy User-Generated Text. Copenhagen, Denmark: Association for Computational Linguistics, 2017, pp. 148-153. [Online]. Available: https://aclanthology.org/W17-4419
L. T. Nguyen and D. Q. Nguyen, "PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsing, " 2021, arXiv:2101.01476.
S. Changpinyo, H. Hu, and F. Sha, "Multi-task learning for sequence tagging: An empirical study, " in Proc. 27th Int. Conf. Comput. Linguistics, Santa Fe, NM, USA, Aug. 2018, pp. 2965-2977. [Online]. Available: https://aclanthology.org/C18-1251
D. Anastasyev, I. Gusev, and E. Indenbom, "Improving part-of-speech tagging via multi-task learning and character-level word representations, " 2018, arXiv:1807.00818.
Y. Deng, W. Zhang, W. Xu, W. Lei, T.-S. Chua, and W. Lam, "A unified multi-task learning framework for multi-goal conversational recommender systems, " 2022, arXiv:2204.06923.
X. Ning and G. Karypis, "Multi-task learning for recommender system, " in Proc. 2nd Asian Conf. Mach. Learn., vol. 13, Tokyo, Japan, M. Sugiyama and Q. Yang, Eds., 2010, pp. 269-284. [Online]. Available: https://proceedings.mlr.press/v13/ning10a.html
Z. Chen et al., "Co-attentive multi-task learning for explainable recommendation, " in Proc. Twenty-Eighth Int. Joint Conf. Artif. Intell., Aug. 2019, pp. 2137-2143, doi: 10.24963/ijcai.2019/296.
S. Ruder, "An overview of multi-task learning in deep neural networks, " 2017, arXiv:1706.05098.
M. Crawshaw, "Multi-task learning with deep neural networks: A survey, " 2020, arXiv:2009.09796.
Y. Zhang and Q. Yang, "A survey on multi-task learning, " IEEE Trans. Knowl. Data Eng., vol. 34, no. 12, pp. 5586-5609, Dec. 2022.
S. Vandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai, and L. Van Gool, "Multi-task learning for dense prediction tasks: A survey, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 7, pp. 3614-3633, Jul. 2022, doi: 10.1109/TPAMI.2021.3054719.
N. Vithayathil Varghese and Q. H. Mahmoud, "A survey of multi-task deep reinforcement learning, " Electronics, vol. 9, no. 9, p. 1363, Aug. 2020. [Online]. Available: https://www.mdpi.com/2079-9292/9/9/1363
Z. Zhang, W. Yu, M. Yu, Z. Guo, and M. Jiang, "A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods, " 2022, arXiv:2204.03508.
S. Chen, Y. Zhang, and Q. Yang, "Multi-task learning in natural language processing: An overview, " 2021, arXiv:2109.09138.
T. Gong et al., "A comparison of loss weighting strategies for multi task learning in deep neural networks, " IEEE Access, vol. 7, pp. 141627-141632, 2019.
A. Zamir, A. Sax, W. Shen, L. Guibas, J. Malik, and S. Savarese, "Taskonomy: Disentangling task transfer learning, " 2018, arXiv:1804.08328.
T. Mensink, J. Uijlings, A. Kuznetsova, M. Gygli, and V. Ferrari, "Factors of influence for transfer learning across diverse appearance domains and task types, " 2021, arXiv:2103.13318.
A. Argyriou, T. Evgeniou, and M. Pontil, "Multi-task feature learning, " in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, vol. 19, B. Scholkopf, J. Platt, and T. Hoffman, Eds., MIT Press, 2006, pp. 1-8. [Online]. Available: https://proceedings.neurips.cc/paper/2006/file/0afa92fc 0f8a9cf051bf2961b06ac56b-Paper.pdf
T. Evgeniou, C. A. Micchelli, and M. Pontil, "Learning multiple tasks with kernel methods, " J. Mach. Learn. Res., vol. 6, no. 21, pp. 615-637, 2005. [Online]. Available: http://jmlr.org/papers/v6/evgeniou05a.html
K. Lounici, M. Pontil, A. B. Tsybakov, and S. van de Geer, "Taking advantage of sparsity in multi-task learning, " 2009, arXiv:0903.1468.
J. Liu, S. Ji, and J. Ye, "Multi-task feature learning via efficient l2, 1-norm minimization, " 2012, arXiv:1205.2631.
S. Ji and J. Ye, "An accelerated gradient method for trace norm minimization, " in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 457-464.
A. Jalali, S. Sanghavi, C. Ruan, and P. Ravikumar, "A dirty model for multi-task learning, " in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, vol. 23, J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, Eds., Curran Associates, 2010, pp. 1-9. [Online]. Available: https://proceedings.neurips. cc/paper/2010/file/00e26af6ac3b1c1c49d7c 3d79c60d000-Paper.pdf
A. Kumar and H. Daume III, "Learning task grouping and overlap in multi-task learning, " 2012, arXiv:1206.6417.
S. Thrun and J. O'Sullivan, Clustering Learning Tasks and the Selective Cross-Task Transfer of Knowledge. Norwell, MA, USA: Kluwer, 1998, pp. 235-257.
Y. Xue, X. Liao, L. Carin, and B. Krishnapuram, "Multi-task learning for classification with Dirichlet process priors, " J. Mach. Learn. Res., vol. 8, no. 2, pp. 35-63, 2007. [Online]. Available: http://jmlr.org/papers/v8/xue07a.html
L. Jacob, F. Bach, and J.-P. Vert, "Clustered multi-task learning: A convex formulation, " 2008, arXiv:0809.2085.
C. Micchelli and M. Pontil, "Kernels for multi-task learning, " in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, vol. 17, L. Saul, Y. Weiss, and L. Bottou, Eds., MIT Press, 2004, pp. 1-8. [Online]. Available: https://proceedings. neurips.cc/paper/2004/file/c4f796afbc626750 1964b46427b3f6ba-Paper.pdf
Q. Gu, Z. Li, and J. Han, "Learning a kernel for multi-task clustering, " in Proc. AAAI Conf. Artif. Intell., Aug. 2011, vol. 25, no. 1, pp. 368-373. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/7914
J. Zhou, J. Chen, and J. Ye, "Clustered multi-task learning via alternating structure optimization, " in Proc. Adv. Neural Inf. Process. Syst., Granada, Spain, vol. 24, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, Eds., Curran Associates, 2011, pp. 1-9. [Online]. Available: https://proceedings. neurips.cc/paper/2011/file/a516a87cfcaef229b342 c437fe2b95f7-Paper.pdf
I. Kokkinos, "UberNet: Training a 'universal' convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory, " 2016, arXiv:1609.02132.
A. Kendall, Y. Gal, and R. Cipolla, "Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, " 2017, arXiv:1705.07115.
F. Heuer, S. Mantowsky, S. S. Bukhari, and G. Schneider, "MultiTask-CenterNet (MCN): Efficient and diverse multitask learning using an anchor free approach, " 2021, arXiv:2108.05060.
D. Xu, W. Ouyang, X. Wang, and N. Sebe, "PAD-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing, " 2018, arXiv:1805.04409.
Z. Zhang, Z. Cui, C. Xu, Y. Yan, N. Sebe, and J. Yang, "Pattern-affinitive propagation across depth, surface normal and semantic segmentation, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2019, pp. 4106-4115.
S. Vandenhende, S. Georgoulis, and L. V. Gool, "MTI-Net: Multi-scale task interaction networks for multi-task learning, " in Proc. 16th Eur. Conf., 2020, pp. 527-543.
D. Bruggemann, M. Kanakis, A. Obukhov, S. Georgoulis, and L. V. Gool, "Exploring relational context for multi-task dense prediction, " in Proc. IEEE/CVF Int. Conf. Comput. Vis., Jun. 2021, pp. 15869-15878.
H. Ye and D. Xu, "InvPT: Inverted pyramid multi-task transformer for dense scene understanding, " 2022, arXiv:2203.07997.
A. Dosovitskiy et al., "An image is worth 16×16 words: Transformers for image recognition at scale, " 2020, arXiv:2010.11929.
S. Ruder, J. Bingel, I. Augenstein, and A. Sogaard, "Latent multi-task architecture learning, " 2017, arXiv:1705.08142.
Y. Gao, J. Ma, M. Zhao, W. Liu, and A. L. Yuille, "NDDR-CNN: Layerwise feature fusing in multi-task CNNs by neural discriminative dimensionality reduction, " 2018, arXiv:1801.08297.
M. Lin, Q. Chen, and S. Yan, "Network in network, " 2013, arXiv:1312.4400.
A. Vaswani et al., "Attention is all you need, " 2017, arXiv:1706.03762.
W. Wang et al., "Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, " 2021, arXiv:2102.12122.
W. Wang et al., "PVT v2: Improved baselines with pyramid vision transformer, " Comput. Vis. Media, vol. 8, no. 3, pp. 415-424, Mar. 2022, doi: 10.1007/s41095-022-0274-8.
Z. Liu et al., "Swin transformer: Hierarchical vision transformer using shifted windows, " 2021, arXiv:2103.14030.
J. Yang et al., "Focal self-attention for local-global interactions in vision transformers, " 2021, arXiv:2107.00641.
R. Hu and A. Singh, "UniT: Multimodal multitask learning with a unified transformer, " 2021, arXiv:2102.10772.
D. Bhattacharjee, T. Zhang, S. Susstrunk, and M. Salzmann, "MulT: An end-to-end multitask learning transformer, " 2022, arXiv:2205.08303.
X. Xu, H. Zhao, V. Vineet, S.-N. Lim, and A. Torralba, "MTFormer: Multi-task learning via transformer and cross-task reasoning, " in Proc. 17th Eur. Conf., Tel Aviv, Israel. Berlin, Germany: Springer-Verlag, Oct. 2022, pp. 304-321, doi: 10.1007/978-3-031-19812-0-18.
Y. Yang and T. Hospedales, "Deep multi-task representation learning: A tensor factorisation approach, " 2016, arXiv:1605.06391.
V. Klema and A. Laub, "The singular value decomposition: Its computation and some applications, " IEEE Trans. Autom. Control, vol. AC-25, no. 2, pp. 164-176, Apr. 1980.
L. R. Tucker, "Some mathematical notes on three-mode factor analysis, " Psychometrika, vol. 31, no. 3, pp. 279-311, Sep. 1966.
Y. Yang and T. M. Hospedales, "Trace norm regularised deep multi-task learning, " 2016, arXiv:1606.04038.
A. A. Rusu et al., "Policy distillation, " 2015, arXiv:1511.06295.
E. Parisotto, J. Lei Ba, and R. Salakhutdinov, "Actor-mimic: Deep multitask and transfer reinforcement learning, " 2015, arXiv:1511.06342.
Y. Whye Teh et al., "Distral: Robust multitask reinforcement learning, " 2017, arXiv:1707.04175.
W.-H. Li and H. Bilen, "Knowledge distillation for multi-task learning, " 2020, arXiv:2007.06889.
G. Ghiasi, B. Zoph, E. D. Cubuk, Q. V. Le, and T.-Y. Lin, "Multi-task self-training for learning general representations, " 2021, arXiv:2108.11353.
X. Yang, J. Ye, and X. Wang, "Factorizing knowledge in neural networks, " 2022, arXiv:2207.03337.
S.-A. Rebuffi, H. Bilen, and A. Vedaldi, "Learning multiple visual domains with residual adapters, " in Proc. Adv. Neural Inf. Process. Syst., vol. 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., Curran Associates, 2017, pp. 1-11. [Online]. Available: https://proceedings.neurips.cc/paper/2017/file/e7b24b112a44fdd9ee93bdf998c6ca0e-Paper.pdf
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition, " 2015, arXiv:1512.03385.
Z. Li and D. Hoiem, "Learning without forgetting, " 2016, arXiv:1606.09282.
S.-A. Rebuffi, H. Bilen, and A. Vedaldi, "Efficient parametrization of multi-domain deep neural networks, " 2018, arXiv:1803.10082.
K.-K. Maninis, I. Radosavovic, and I. Kokkinos, "Attentive single-tasking of multiple tasks, " 2019, arXiv:1904.08918.
J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, "Squeeze-and-excitation networks, " 2017, arXiv:1709.01507.
J. Pfeiffer, A. Kamath, A. Ruckle, K. Cho, and I. Gurevych, "AdapterFusion: Non-destructive task composition for transfer learning, " 2020, arXiv:2005.00247.
A. Cooper Stickland and I. Murray, "BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning, " 2019, arXiv:1902.02671.
E. Meyerson and R. Miikkulainen, "Beyond shared hierarchies: Deep multitask learning through soft layer ordering, " 2017, arXiv:1711.00108.
J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi, "Modeling task relationships in multi-task learning with multi-gate mixture-of-experts, " in Proc. 24th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, Jul. 2018, pp. 1930-1939.
N. Shazeer et al., "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, " 2017, arXiv:1701.06538.
H. Hazimeh et al., "DSelect-k: Differentiable selection in the mixture of experts with applications to multi-task learning, " 2021, arXiv:2106.03760.
C. Fernando et al., "PathNet: Evolution channels gradient descent in super neural networks, " 2017, arXiv:1701.08734.
J. Liang, E. Meyerson, and R. Miikkulainen, "Evolutionary architecture search for deep multitask networks, " 2018, arXiv:1803.03745.
A. Gesmundo and J. Dean, "An evolutionary approach to dynamic introduction of tasks in large-scale multitask learning systems, " 2022, arXiv:2205.12755.
K. Maziarz, E. Kokiopoulou, A. Gesmundo, L. Sbaiz, G. Bartok, and J. Berent, "Flexible multi-task networks by learning parameter allocation, " 2019, arXiv:1910.04915.
X. Sun, R. Panda, R. Feris, and K. Saenko, "AdaShare: Learning what to share for efficient deep multi-task learning, " 2019, arXiv:1911.12423.
E. Jang, S. Gu, and B. Poole, "Categorical reparameterization with gumbel-softmax, " 2016, arXiv:1611.01144.
L. Zhang, X. Liu, and H. Guan, "AutoMTL: A programming framework for automating efficient multi-task learning, " 2021, arXiv:2110.13076.
F. J. S. Bragman, R. Tanno, S. Ourselin, D. C. Alexander, and M. J. Cardoso, "Stochastic filter groups for multi-task CNNs: Learning specialist and generalist convolution kernels, " 2019, arXiv:1908.09597.
D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, "Variational inference: A review for statisticians, " J. Amer. Stat. Assoc., vol. 112, no. 518, pp. 859-877, Apr. 2017, doi: 10.1080/01621459.2017.1285773.
Y. Lu, A. Kumar, S. Zhai, Y. Cheng, T. Javidi, and R. Feris, "Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification, " 2016, arXiv:1611.05377.
S. Vandenhende, S. Georgoulis, B. De Brabandere, and L. Van Gool, "Branched multi-task networks: Deciding what layers to share, " 2019, arXiv:1904.02920.
T. Vu, Y. Zhou, C. Wen, Y. Li, and J.-M. Frahm, "Toward edge-efficient dense predictions with synergistic multi-task neural architecture search, " 2022, arXiv:2210.01384.
H. Benmeziane, K. El Maghraoui, H. Ouarnoughi, S. Niar, M. Wistuba, and N. Wang, "A comprehensive survey on hardware-aware neural architecture search, " 2021, arXiv:2101.09336.
H. Hu, D. Dey, M. Hebert, and J. A. Bagnell, "Learning anytime predictions in neural networks via adaptive loss balancing, " 2017, arXiv:1708.06832.
L. Liebel and M. Korner, "Auxiliary tasks in multi-task learning, " 2018, arXiv:1805.06334.
L. Liu et al., "Towards impartial multi-task learning, " in Proc. Int. Conf. Learn. Represent., 2021. [Online]. Available: https://openreview.net/forum?id=IMPnRXEWpvr
S. Chennupati, G. Sistu, S. Yogamani, and S. A. Rawashdeh, "MultiNet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning, " 2019, arXiv:1904.08492.
S. Liu, E. Johns, and A. J. Davison, "End-to-end multi-task learning with attention, " 2018, arXiv:1803.10704.
B. Lin, F. Ye, Y. Zhang, and I. W. Tsang, "Reasonable effectiveness of random weighting: A litmus test for multi-task learning, " 2021, arXiv:2111.10603.
Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, "GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks, " 2017, arXiv:1711.02257.
Z. Chen et al., "Just pick a sign: Optimizing deep multitask models with gradient sign dropout, " 2020, arXiv:2010.06808.
Y. Du, W. M. Czarnecki, S. M. Jayakumar, M. Farajtabar, R. Pascanu, and B. Lakshminarayanan, "Adapting auxiliary losses using gradient similarity, " 2018, arXiv:1812.02224.
M. Suteu and Y. Guo, "Regularizing deep multi-task networks using orthogonal gradients, " 2019, arXiv:1912.06844.
T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn, "Gradient surgery for multi-task learning, " 2020, arXiv:2001.06782.
Z. Wang, Y. Tsvetkov, O. Firat, and Y. Cao, "Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models, " in Proc. Int. Conf. Learn. Represent., 2021, pp. 1-22. [Online]. Available: https://openreview.net/forum?id=F1vEjWK-lH
A. Javaloy and I. Valera, "RotoGrad: Gradient homogenization in multitask learning, " 2021, arXiv:2103.02631.
X. Lin, Z. Yang, Q. Zhang, and S. Kwong, "Controllable Pareto multi-task learning, " 2020, arXiv:2010.06313.
A. Navon, A. Shamsian, G. Chechik, and E. Fetaya, "Learning the Pareto front with hypernetworks, " 2021, arXiv:2010.04104.
J.-A. Desideri, "Multiple-gradient descent algorithm (MGDA) for multiobjective optimization, " Comp. Rendus. Mathematique, vol. 350, nos. 5-6, pp. 313-318, Mar. 2012.
O. Sener and V. Koltun, "Multi-task learning as multi-objective optimization, " 2018, arXiv:1810.04650.
M. Jaggi, "Revisiting frank-wolfe: Projection-free sparse convex optimization, " in Proc. 30th Int. Conf. Mach. Learn., vol. 28, Atlanta, Georgia, S. Dasgupta and D. McAllester, Eds., Jun. 2013, pp. 427-435. [Online]. Available: https://proceedings.mlr.press/v28/jaggi13.html
B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu, "Conflict-averse gradient descent for multi-task learning, " 2021, arXiv:2110.14048.
X. Lin, H.-L. Zhen, Z. Li, Q. Zhang, and S. Kwong, "Pareto multi-task learning, " 2019, arXiv:1912.12854.
H.-L. Liu, F. Gu, and Q. Zhang, "Decomposition of a multiobjective optimization problem into a number of simple multiobjective subproblems, " IEEE Trans. Evol. Comput., vol. 18, no. 3, pp. 450-455, Jun. 2014.
P. Ma, T. Du, and W. Matusik, "Efficient continuous Pareto exploration in multi-task learning, " 2020, arXiv:2006.16434.
D. Ha, A. Dai, and Q. V. Le, "HyperNetworks, " 2016, arXiv:1609.09106.
M. Momma, C. Dong, and J. Liu, "A multi-objective/multi-task learning framework induced by Pareto stationarity, " in Proc. 39th Int. Conf. Mach. Learn., vol. 162, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Eds., 2022, pp. 15895-15907. [Online]. Available: https://proceedings. mlr.press/v162/momma22a.html
A. Navon et al., "Multi-task learning as a bargaining game, " 2022, arXiv:2202.01017.
J. Nash, "Two-person cooperative games, " Econometrica, vol. 21, no. 1, pp. 128-140, Jan. 1953. [Online]. Available: http://www.jstor.org/stable/1906951
D. Xin, B. Ghorbani, J. Gilmer, A. Garg, and O. Firat, "Do current multi-task optimization methods in deep learning even help?" in Proc. Adv. Neural Inf. Process. Syst., A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022, pp. 1-13. [Online]. Available: https://openreview.net/forum?id=A2Ya5aLtyuG
Z. Zhang, P. Luo, C. C. Loy, and X. Tang, "Facial landmark detection by deep multi-task learning, " in Proc. 13th Eur. Conf., Sep. 2014, pp. 94-108.
J. Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lee, "12-in-1: Multi-task vision and language representation learning, " 2019, arXiv:1912.02315.
C. Li, J. Yan, F. Wei, W. Dong, Q. Liu, and H. Zha, "Self-paced multi-task learning, " 2016, arXiv:1604.01474.
M. Guo, A. Haque, D.-A. Huang, S. Yeung, and L. Fei-Fei, "Dynamic task prioritization for multitask learning, " in Proc. Eur. Conf. Comput. Vis. (ECCV), Sep. 2018, pp. 270-287.
T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, "Focal loss for dense object detection, " 2017, arXiv:1708.02002.
S. Sharma, A. Jha, P. Hegde, and B. Ravindran, "Learning to multi-task by active sampling, " 2017, arXiv:1702.06053.
Z. Kang, K. Grauman, and F. Sha, "Learning with whom to share in multi-task feature learning, " in Proc. 28th Int. Conf. Mach. Learn. (ICML). Madison, WI, USA: Omnipress, 2011, pp. 521-528.
M. Long, Z. Cao, J. Wang, and P. S. Yu, "Learning multiple tasks with multilinear relationship networks, " in Proc. 31st Int. Conf. Neural Inf. Process. Syst. Red Hook, NY, USA: Curran Associates, 2017, pp. 1593-1602.
Y. Zhang and D.-Y. Yeung, "A regularization approach to learning task relationships in multitask learning, " ACM Trans. Knowl. Discovery Data, vol. 8, no. 3, pp. 1-31, Jun. 2014.
M. Ohlson, M. R. Ahmad, and D. von Rosen, "The multilinear normal distribution: Introduction and some basic properties, " J. Multivariate Anal., vol. 113, pp. 37-47, Jan. 2013. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0047259X11001047
K. Dwivedi and G. Roig, "Representation similarity analysis for efficient task taxonomy & transfer learning, " 2019, arXiv:1904.11740.
T. Standley, A. R. Zamir, D. Chen, L. Guibas, J. Malik, and S. Savarese, "Which tasks should be learned together in multi-task learning?" 2019, arXiv:1905.07553.
C. Fifty, E. Amid, Z. Zhao, T. Yu, R. Anil, and C. Finn, "Efficiently identifying task groupings for multi-task learning, " 2021, arXiv:2109.04617.
C. Doersch and A. Zisserman, "Multi-task self-supervised visual learning, " 2017, arXiv:1708.07860.
C. Doersch, A. Gupta, and A. A. Efros, "Unsupervised visual representation learning by context prediction, " 2015, arXiv:1505.05192.
R. Zhang, P. Isola, and A. A. Efros, "Colorful image colorization, " 2016, arXiv:1603.08511.
A. Dosovitskiy, P. Fischer, J. Tobias Springenberg, M. Riedmiller, and T. Brox, "Discriminative unsupervised feature learning with exemplar convolutional neural networks, " 2014, arXiv:1406.6909.
D. Pathak, R. Girshick, P. Dollar, T. Darrell, and B. Hariharan, "Learning features by watching objects move, " 2016, arXiv:1612.06370.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2009, pp. 248-255.
W. Lee, J. Na, and G. Kim, "Multi-task self-supervised object detection via recycling of bounding box annotations, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 4979-4988.
J. Cho, Y. Kim, H. Jung, C. Oh, J. Youn, and K. Sohn, "Multi-task self-supervised visual representation learning for monocular road segmentation, " in Proc. IEEE Int. Conf. Multimedia Expo (ICME), Oct. 2018, pp. 1-6.
A. Geiger, P. Lenz, and R. Urtasun, "Are we ready for autonomous driving? The KITTI vision benchmark suite, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2012, pp. 3354-3361.
H. Hirschmuller, "Accurate and efficient stereo processing by semi-global matching and mutual information, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), vol. 2, Jun. 2005, pp. 807-814.
D. Hernandez-Juarez, A. Espinosa, D. Vazquez, A. M. Lopez, and J. C. Moure, "GPU-accelerated real-time stixel computation, " 2016, arXiv:1610.04124.
N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, "Indoor segmentation and support inference from RGBD images, " in Proc. 12th Eur. Conf. Comput. Vis., vol. 7576, Oct. 2012, pp. 746-760.
J. Pfister, K. Kobs, and A. Hotho, "Self-supervised multi-task pretraining improves image aesthetic assessment, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2021, pp. 816-825.
M.-I. Georgescu, A. Barbalau, R. T. Ionescu, F. S. Khan, M. Popescu, and M. Shah, "Anomaly detection in video via self-supervised and multi-task learning, " 2020, arXiv:2011.07491.
A. Barbalau et al., "SSMTL++: Revisiting self-supervised multi-task learning for video anomaly detection, " 2022, arXiv:2207.08003.
J. Lu, D. Batra, D. Parikh, and S. Lee, "ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks, " 2019, arXiv:1908.02265.
A. Balajee Vasudevan, D. Dai, and L. Van Gool, "Sound and visual representation learning with multiple pretraining tasks, " 2022, arXiv:2201.01046.
R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir, "MultiMAE: Multi-modal multi-task masked autoencoders, " 2022, arXiv:2204.01678.
K. He, X. Chen, S. Xie, Y. Li, P. Dollar, and R. Girshick, "Masked autoencoders are scalable vision learners, " 2021, arXiv:2111.06377.
R. Ranftl, A. Bochkovskiy, and V. Koltun, "Vision transformers for dense prediction, " 2021, arXiv:2103.13413.
B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, "Masked-attention mask transformer for universal image segmentation, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 1280-1289.
T.-Y. Lin et al., "Microsoft COCO: Common objects in context, " 2014, arXiv:1405.0312.
Z. Tu, Q. Zhou, H. Zou, and X. Zhang, "A multi-task dense network with self-supervised learning for retinal vessel segmentation, " Electronics, vol. 11, no. 21, p. 3538, 2022. [Online]. Available: https://www.mdpi.com/2079-9292/11/21/3538
W. Liao et al., "MUSCLE: Multi-task self-supervised continual learning to pre-train deep models for X-ray images of multiple body parts, " in Medical Image Computing and Computer Assisted Intervention-MICCAI, L. Wang, Q. Dou, P. T. Fletcher, S. Speidel, and S. Li, Eds., Cham, Switzerland: Springer, 2022, pp. 151-161.
L. Chaves, A. Bissoto, E. Valle, and S. Avila, "An evaluation of self-supervised pre-training for skin-lesion analysis, " 2021, arXiv:2106.09229.
H.-H. Wu et al., "Multi-task self-supervised pre-training for music classification, " 2021, arXiv:2102.03229.
B. Li, Y. Weng, F. Xia, S. He, B. Sun, and S. Li, "LingJing at SemEval-2022 task 1: Multi-task self-supervised pre-training for multilingual reverse dictionary, " in Proc. 16th Int. Workshop Semantic Eval., 2022, pp. 29-35. [Online]. Available: https://aclanthology.org/2022.semeval-1.4
F. Wang, X. Wang, and T. Li, "Semi-supervised multi-task learning with task regularizations, " in Proc. 9th IEEE Int. Conf. Data Mining, Dec. 2009, pp. 562-568.
H. Jiang, E. Learned-Miller, G. Larsson, M. Maire, and G. Shakhnarovich, "Self-supervised relative depth learning for urban scene understanding, " 2017, arXiv:1712.04850.
M. Klingner, J.-A. Termohlen, J. Mikolajczyk, and T. Fingscheidt, "Self-supervised monocular depth estimation: Solving the dynamic object problem by semantic guidance, " 2020, arXiv:2007.06936.
J. Novosel, "Boosting semantic segmentation with multi-task self-supervised learning for autonomous driving applications, " in Proc. NIPS, 2019, pp. 1-11.
L. Hoyer, D. Dai, Y. Chen, A. Koring, S. Saha, and L. Van Gool, "Three ways to improve semantic segmentation with self-supervised depth estimation, " 2020, arXiv:2012.10782.
S. Yun, D. Han, S. Joon Oh, S. Chun, J. Choe, and Y. Yoo, "CutMix: Regularization strategy to train strong classifiers with localizable features, " 2019, arXiv:1905.04899.
V. Olsson, W. Tranheden, J. Pinto, and L. Svensson, "ClassMix: Segmentation-based data augmentation for semi-supervised learning, " 2020, arXiv:2007.07936.
A. Tarvainen and H. Valpola, "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results, " 2017, arXiv:1703.01780.
L. Hoyer, D. Dai, Q. Wang, Y. Chen, and L. Van Gool, "Improving semi-supervised and domain-adaptive semantic segmentation with self-supervised depth estimation, " 2021, arXiv:2108.12545.
L. Gao, C. Khamesra, U. Kumbhar, and A. Aglawe, "Multi-task self-supervised learning for image segmentation task, " 2023, arXiv:2302.02483.
I. J. Goodfellow et al., "Generative adversarial networks, " 2014, arXiv:1406.2661.
J. Donahue, P. Krahenbuhl, and T. Darrell, "Adversarial feature learning, " 2016, arXiv:1605.09782.
W.-C. Hung, Y.-H. Tsai, Y.-T. Liou, Y.-Y. Lin, and M.-H. Yang, "Adversarial learning for semi-supervised semantic segmentation, " 2018, arXiv:1802.07934.
J. Zhang, Z. Li, C. Zhang, and H. Ma, "Robust adversarial learning for semi-supervised semantic segmentation, " in Proc. IEEE Int. Conf. Image Process. (ICIP), Oct. 2020, pp. 728-732.
R. Hadsell, S. Chopra, and Y. LeCun, "Dimensionality reduction by learning an invariant mapping, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), vol. 2, Jun. 2006, pp. 1735-1742.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, "Momentum contrast for unsupervised visual representation learning, " 2019, arXiv:1911.05722.
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, "Context encoders: Feature learning by inpainting, " 2016, arXiv:1604.07379.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations, " 2020, arXiv:2002.05709.
N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, "Indoor segmentation and support inference from RGBD images, " in Proc. ECCV, Florence, Italy, 2012.
P. Bilinski and V. Prisacariu, "Dense decoder shortcut connections for single-pass semantic segmentation, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 6596-6605.
L. Yu, Y. Gao, J. Zhou, J. Zhang, and Q. Wu, "Multi-layer feature aggregation for deep scene parsing models, " 2020, arXiv:2011.02572.
S. Borse, Y. Wang, Y. Zhang, and F. Porikli, "InverseForm: A loss function for structured boundary-aware segmentation, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2021, pp. 5901-5911.
L. Piccinelli et al., "UniDepth: Universal monocular metric depth estimation, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2024, pp. 10106-10116.
L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, "Depth anything: Unleashing the power of large-scale unlabeled data, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2024, pp. 10371-10381.
S. Hickson, K. Raveendran, A. Fathi, K. Murphy, and I. Essa, "Floors are flat: Leveraging semantics for real-time surface normal prediction, " in Proc. IEEE/CVF Int. Conf. Comput. Vis. Workshop, Jun. 2019, pp. 4065-4074.
G. Bae, I. Budvytis, and R. Cipolla, "Estimating and exploiting the aleatoric uncertainty in surface normal estimation, " in Proc. IEEE/CVF Int. Conf. Comput. Vis., Jun. 2021, pp. 13137-13146.
L. Piccinelli, C. Sakaridis, and F. Yu, "iDisc: Internal discretization for monocular depth estimation, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2023, pp. 21477-21487.
L. Zhou et al., "Pattern-structure diffusion for multi-task learning, " in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Los Alamitos, CA, USA, Jun. 2020, pp. 4513-4522, doi: 10.1109/CVPR42600.2020.00457.
H. Ye and D. Xu, "Taskprompter: Spatial-channel multi-task prompting for dense scene understanding, " in Proc. 11th Int. Conf. Learn. Represent., 2023, pp. 1-20. [Online]. Available: https://openreview.net/forum?id=-CwPopPJda
Y. Xu, Y. Yang, and L. Zhang, "DeMT: Deformable mixer transformer for multi-task learning of dense prediction, " in Proc. AAAI Conf. Artif. Intell., 2023, pp. 3072-3080.
Y. Li, J. Hu, J. Sun, S. Zhao, Q. Zhang, and Y. Lin, "A novel multi-task self-supervised representation learning paradigm, " in Proc. IEEE Int. Conf. Artif. Intell. Ind. Design (AIID), May 2021, pp. 94-99.
J.-B. Grill et al., "Bootstrap your own latent: A new approach to self-supervised learning, " in Proc. Adv. Neural Inf. Process. Syst., 2020, pp. 21271-21284.
A. Zamir et al., "Robust learning through cross-task consistency, " 2020, arXiv:2006.04096.
E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville, "FiLM: Visual reasoning with a general conditioning layer, " 2017, arXiv:1709.07871.
B. McCann, N. S. Keskar, C. Xiong, and R. Socher, "The natural language decathlon: Multitask learning as question answering, " 2018, arXiv:1806.08730.
N. Simard and G. Lagrange, "Improving few-shot learning with auxiliary self-supervised pretext tasks, " 2021, arXiv:2101.09825.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, "The PASCAL visual object classes (VOC) challenge, " Int. J. Comput. Vis., vol. 88, no. 2, pp. 303-338, Jun. 2010.
D. Kim, J. Kim, S. Cho, C. Luo, and S. Hong, "Universal few-shot learning of dense prediction tasks with visual token matching, " in Proc. 11th Int. Conf. Learn. Represent., 2023, pp. 1-26. [Online]. Available: https://openreview.net/forum?id=88nT0j5jAn
M. Cordts et al., "The cityscapes dataset for semantic urban scene understanding, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2016, pp. 3213-3223.
J. M. Alvarez, T. Gevers, Y. Lecun, and A. Lpez, "Road scene segmentation from a single image, " in Proc. 12th Eur. Conf. Comput. Vis., Oct. 2012, pp. 376-389.
R. Zhang, S. A. Candra, K. Vetter, and A. Zakhor, "Sensor fusion for semantic segmentation of urban scenes, " in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), May 2015, pp. 1850-1857.
G. Ros, S. Ramos, M. Granados, A. Bakhtiary, D. Vazquez, and A. M. Lopez, "Vision-based offline-online perception paradigm for autonomous driving, " in Proc. IEEE Winter Conf. Appl. Comput. Vis., Jan. 2015, pp. 231-238.
Y. Hong, J. Wang, W. Sun, and H. Pan, "Minimalist and high-performance semantic segmentation with plain vision transformers, " 2023, arXiv:2310.12755.
B. Lin and Y. Zhang, "LibMTL: A Python library for multi-task learning, " 2022, arXiv:2203.14338.
