Data input; Data output; Enterprise analytics; Input and outputs; Input datas; Large scale simulations; Parameter data; Potential sources; Root cause; Software updates; Information Systems; Hardware and Architecture
Abstract :
[en] Applications in domains ranging from large-scale simulations in astrophysics and biology to enterprise analytics rely on computational pipelines. A pipeline consists of modules and their associated parameters, data inputs, and outputs, which are orchestrated to produce a set of results. If some modules derive unexpected outputs, the pipeline can crash or lead to incorrect results. Debugging these pipelines is difficult since there are many potential sources of errors including: bugs in the code, input data, software updates, and improper parameter settings. We present BugDoc, a system that automatically infers the root causes and derive succinct explanations of failures for black-box pipelines. BugDoc does so by using provenance from previous runs of a given pipeline to derive hypotheses for the errors, and then iteratively runs new pipeline configurations to test these hypotheses. Besides identifying issues associated with computational modules in a pipeline, we also propose methods for: “opportunistic group testing” to identify portions of data inputs that might be responsible for failed executions (what we call), helping users narrow down the cause of failure; and “selective instrumentation” to determine nodes in pipelines that should be instrumented to improve efficiency and reduce the number of iterations to test. Through a case study of deployed workflows at a software company and an experimental evaluation using synthetic pipelines, we assess the effectiveness of BugDoc and show that it requires fewer iterations to derive root causes and/or achieves higher quality results than previous approaches.
Disciplines :
Computer science
Author, co-author :
DE PAULA LOURENCO, Raoni ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
Freire, Juliana; Tandon School of Engineering, New York University, Brooklyn, United States
Simon, Eric; SAP, Levallois-Perret, France
Weber, Gabriel; Amazon, São Paulo, Brazil
Shasha, Dennis; Courant Institute of Mathematical Sciences, New York University, New York, United States
External co-authors :
yes
Language :
English
Title :
BugDoc: Iterative debugging and explanation of pipeline
Publication date :
2023
Journal title :
VLDB Journal
ISSN :
1066-8888
eISSN :
0949-877X
Publisher :
Springer Science and Business Media Deutschland GmbH
National Science Foundation Conselho Nacional de Desenvolvimento Científico e Tecnológico Defense Advanced Research Projects Agency
Funding text :
We thank the Data X-Ray and Explanation Tables authors for sharing their code with us. We are also grateful to Fernando Chirigati, Neel Dey, and Peter Bailis for providing the real-world pipelines. This work has been supported in part by NSF grants IIS-1916505, IIS-2106888, IOS-1339362, MCB-1158273, MCB-1412232, and OAC-1934464; CNPq (Brazil) grant 209623/2014-4; the DARPA D3M program; and NYU WIRELESS. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of funding agencies.
Artho, C.: Iterative delta debugging. Int. J. Softw. Tools Technol. Transfer 13, 223–246 (2010) DOI: 10.1007/s10009-010-0139-9
Attariyan, M., Flinn, J.: Automating configuration troubleshooting with confaid. login 36(1), 1–14 (2011)
Attariyan, M., Chow, M., Flinn, J.: X-ray: Automating root-cause diagnosis of performance anomalies in production software. In: Proceedings of USENIX OSDI, pp 307–320 (2012)
Bailis, P., Gan, E., Madden, S., Narayanan, D., Rong, K., Suri, S.: Macrobase: Prioritizing attention in fast data. In: Proceedings of ACM SIGMOD, pp 541–556 (2017)
Bala, A., Chana, I.: Intelligent failure prediction models for scientific workflows. Expert Syst. Appl. 3, 980–989 (2015) DOI: 10.1016/j.eswa.2014.09.014
Berger, B., Rompel, J., Shor, P.W.: Efficient NC algorithms for set cover with applications to learning and geometry. J. Computer Syst. Sci. 49, 454–477 (1994) DOI: 10.1016/S0022-0000(05)80068-6
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. JMLR 13, 281–305 (2012)
Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Proceedings of NIPS, pp 2546–2554 (2011)
Bergstra, J., Yamins, D., Cox, DD.: Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In: Proceedings of ICML, pp 115–123 (2013)
Bleifuß T, Kruse S., Naumann, F.: Efficient denial constraint discovery with hydra. Proceedings of VLDB Endowment 3, 311–323 (2017)
Brock, A., Donahue, J., Simonyan, K.: Large scale gan training for high fidelity natural image synthesis. arxiv:1809.11096 (2018)
Chen, A., Wu, Y., Haeberlen, A., Loo, BT., Zhou, W.: Data provenance at internet scale: Architecture, experiences, and the road ahead. In: Proceedings of CIDR, pp 1–7 (2017)
Chen, MY., Kiciman, E., Fratkin, E., Fox, A., Brewer, E.: Pinpoint: Problem determination in large, dynamic internet services. In: Proceedings of IEEE DSN, pp 595–604 (2002)
Chirigati, F., Doraiswamy, H., Damoulas, T., Freire, J.: Data polygamy: The many-many relationships among urban spatio-temporal data sets. In: Proceedings of ACM SIGMOD, pp 1011–1025 (2016)
Colbourn, C.J., Martirosyan, S.S., Mullen, G.L., Shasha, D., Sherwood, G.B., Yucas, J.L.: Products of mixed covering arrays of strength two. J. Comb. Des. 2, 124–138 (2006) DOI: 10.1002/jcd.20065
Dalibard, V., Schaarschmidt, M., Yoneki, E.: Boat: Building auto-tuners with structured bayesian optimization. In: Proceedings of WWW, pp 479–488 (2017)
Dolatnia, N., Fern, A., Fern, X.: Bayesian Optimization with Resource Constraints and Production. In: Proceedings of ICAPS, pp 115–123 (2016)
El Gebaly, K., Agrawal, P., Golab, L., Korn, F., Srivastava, D.: Interpretable and informative explanations of outcomes. Proceedings of VLDB Endowment 1, 61–72 (2014)
Godefroid, P., Levin, MY., Molnar, DA.: Automated whitebox fuzz testing. In: Proceedings of NDSS, p 151-166 (2008)
Goodfellow, IJ., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. arxiv:1406.2661 (2014)
Gudmundsdottir, H., Salimi, B., Balazinska, M., Ports, DR., Suciu, D.: A demonstration of interactive analysis of performance measurements with viska. In: Proceedings of ACM SIGMOD, pp 1707–1710 (2017)
Gulzar, MA., Wang, S., Kim, M.: Bigsift: Automated debugging of big data analytics in data-intensive scalable computing. In: Proceedings of ESEC/FSE, pp 863–866 (2018)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. arxiv:1706.08500 (2017)
Holler, C., Herzig, K., Zeller, A.: Fuzzing with code fragments. In: Proceedings of USENIX Security Symposium, pp 445–458 (2012)
Huang, J.: Programing implementation of the quine-mccluskey method for minimization of boolean expression. CoRR pp 1–22, arxiv:1410.1059 (2014)
Hutter, F., Hoos, HH., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Proceedings of LION-5, pp 507–523 (2011)
Jouppi, N.P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., Boyle, R., Pl, Cantin, Chao, C., Clark, C., Coriell, J., Daley, M., Dau, M., Dean, J., Gelb, B., Ghaemmaghami, T.V., Gottipati, R., Gulland, W., Hagmann, R., Ho, C.R., Hogberg, D., Hu, J., Hundt, R., Hurt, D., Ibarz, J., Jaffey, A., Jaworski, A., Kaplan, A., Khaitan, H., Killebrew, D., Koch, A., Kumar, N., Lacy, S., Laudon, J., Law, J., Le, D., Leary, C., Liu, Z., Lucke, K., Lundin, A., MacKean, G., Maggiore, A., Mahony, M., Miller, K., Nagarajan, R., Narayanaswami, R., Ni, R., Nix, K., Norrie, T., Omernick, M., Penukonda, N., Phelps, A., Ross, J., Ross, M., Salek, A., Samadiani, E., Severn, C., Sizikov, G., Snelham, M., Souter, J., Steinberg, D., Swing, A., Tan, M., Thorson, G., Tian, B., Toma, H., Tuttle, E., Vasudevan, V., Walter, R., Wang, W., Wilcox, E., Yoon, D.H.: In-datacenter performance analysis of a tensor processing unit. SIGARCH Computer Arch. News 2, 1–12 (2017)
Krishna, R., Iqbal, MS., Javidian, MA., Ray, B., Jamshidi, P.: Cadet: A systematic method for debugging misconfigurations using counterfactual reasoning. arxiv:2010.06061 (2020)
Krizhevsky, A., et al.: Learning multiple layers of features from tiny images. Tech. rep, Citeseer (2009)
Lee, KW., Pedarsani, R., Ramchandran, K.: Saffron: A fast, efficient, and robust framework for group testing based on sparse-graph codes. IEEE Transactions Signal Process. pp 1–10 (2015)
Lewis, D.: Counterfactuals. Wiley (2013)
Liblit, B., Naik, M., Zheng, A.X., Aiken, A., Jordan, M.I.: Scalable statistical bug isolation. In Proceedings of ACM SIGPLAN 6, 15–26 (2005)
Liu, FT., Ting, KM., Zhou, ZH.: Isolation forest. In: 2008 eighth ieee international conference on data mining, IEEE, pp 413–422 (2008)
Lourenço, R., Freire, J., Shasha, D.: Debugging machine learning pipelines. In: Proceedings of DEEM (2019)
Lourenço, R., Freire, J., Shasha, D.: Bugdoc: A system for debugging computational pipelines. In: Proceedings of ACM SIGMOD (Demo) (2020a)
Lourenço, R., Freire, J., Shasha, D.: Bugdoc: Algorithms to debug computational processes. In: Proceedings of ACM SIGMOD (2020b)
Lucic, M., Kurach, K., Michalski, M., Gelly, S., Bousquet, O.: Are gans created equal? a large-scale study. arxiv:1711.10337 (2017)
Macula, A.J., Popyack, L.J.: A group testing method for finding patterns in data. Discrete Appl. Math. 1–2, 149–157 (2004) DOI: 10.1016/j.dam.2003.07.009
Meliou, A., Roy, S., Suciu, D.: Causality and explanations in databases. PVLDB 13, 1715–1716 (2014)
Pearl, J.: Causality: Models, Reasoning and Inference, 2nd edn. Cambridge University Press (2009)
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arxiv:1511.06434 (2015)
Rezig, EK., Cao, L.,, Simonini, G., Schoemans, M., Madden, S., Tang, N., Ouzzani, M., Stonebraker, M.: Dagger: A data (not code) debugger. In: CIDR 2020, 10th Conference on Innovative Data Systems Research, Amsterdam, The Netherlands, January 12-15, 2020, Online Proceedings, www.cidrdb.org, http://cidrdb.org/cidr2020/papers/p35-rezig-cidr20.pdf (2020)
Snoek, J., Larochelle, H., Adams, RP.: Practical bayesian optimization of machine learning algorithms. In: Proceedings of NIPS, pp 2951–2959 (2012)
Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, MMA., Prabhat, P., Adams, RP.: Scalable bayesian optimization using deep neural networks. In: Proceedings of the ICML, pp 2171–2180 (2015)
Van Aken, D., Pavlo, A., Gordon, GJ., Zhang, B.: Automatic database management system tuning through large-scale machine learning. In: Proceedings of ACM SIGMOD, pp 1009–1024 (2017)
Wang, X., Dong, XL., Meliou, A.: Data x-ray: A diagnostic tool for data errors. In: Proceedings of ACM SIGMOD, pp 1231–1245 (2015)
Wang, X., Meliou, A., Wu, E.: Qfix: Diagnosing errors through query histories. In: Proceedings of ACM SIGMOD, pp 1369–1384 (2017)
Yoon, DY., Niu, N., Mozafari, B.: Dbsherlock: A performance diagnostic tool for transactional databases. In: Proceedings of ACM SIGMOD, pp 1599–1614 (2016)
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D.: Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proceeding of ICCV, pp 5908–5916 (2017)
Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. arxiv:1805.08318 (2018)
Zheng, AX., Jordan, MI., Liblit, B., Naik, M., Aiken, A.: Statistical debugging: Simultaneous identification of multiple bugs. In: Proceedings of ICML, pp 1105–1112 (2006)
