Comparison of Machine Learning Methods: Conclusions and Future Work, and References

Table of Links

• Abstract and Introduction
• Background
• Approach
• Experiments and Results
• Discussion and Qualitative Analysis
• Related Work
• Conclusions and Future Work, and References

7 Conclusions and Future Work

This study focuses on automated issue assignment using proprietary issue reports obtained from the electronic product manufacturer’s issue tracking system. The objective of the issue assignment approach is to assign issues to appropriate team members based on their respective fields. The team members are categorized into Software Developer, Software Tester, Team Leader, and UI/UX Designer. Among these categories, the majority of the data set consists of developers. Efficiently allocating issues to developers is critical for effective time management. To achieve this, we further classify developers into Senior, Mid, and Junior levels, which are widely accepted labels in the industry.

Our focus lies in extracting features from the filled Jira columns, as well as the title and description texts of the issues, utilizing NLP techniques. These features serve as inputs to our learning methods, enabling us to analyze and classify the issues effectively. Additionally, we employ other commonly used word embedding methods which are Tf-Idf, BOW, and Word2Vec to generate feature vectors from the text fields. This step, implemented using the Sklearn and Gensim library, allows us to compare the performance of our feature set against alternative approaches. Furthermore, to assess the effectiveness of our overall methodology, we incorporate widely adopted deep-learning techniques, namely DistilBert, Roberta, and Electra.

Following the production of feature vectors, we proceed to implement the proposed system utilizing established machine learning techniques. With the aim of enhancing predictive performance, we employ ensemble methods that leverage a diverse range of machine-learning algorithms. To evaluate the effectiveness of our system, we employ widely recognized metrics such as accuracy, precision, recall, and F1-score which serve as indicators of its performance. To further refine our predictions, we employ a robust technique known as 10-fold crossvalidation. In order to conduct a thorough statistical analysis, we construct a matrix to compare and contrast the effectiveness of our proposed strategies. This matrix allows us to assess the performance of our system across different algorithms, ensemble techniques, and evaluation metrics.

Our future endeavors involve the development of a versatile tool applicable to diverse software team models. To fortify our work, we actively engage in discussions and pursue collaborations to acquire data sets from businesses operating across various domains, such as game development and banking applications. This broadened data set will enable us to enhance our model’s capabilities for multi-class classification, accommodating different roles within software teams, including product owners and business analysts. Furthermore, we are committed to ensuring compatibility and flexibility by incorporating various business branches into our data set. By incorporating real-world data obtained directly from industry sources, both in English and Turkish, we will conduct comprehensive evaluations through diverse studies. Expanding on the existing features, we intend to utilize the same data set for future research endeavors, such as effort estimation [53,60], further solidifying the value and applicability of our work in the field.

References

1. Aktas, E.U., Yeniterzi, R., Yilmaz, C.: Turkish issue report classification in banking domain. In: 28th Signal Processing and Communications Applications Conference (SIU). pp. 1–4. Gaziantep, Turkey (2020)

1. Alenezi, M., Magel, K., Banitaan, S.: Efficient bug triaging using text mining. Journal of Software 8(9), 2185–2190 (2013)

2. Aljarah, I., Banitaan, S., Abufardeh, S., Jin, W., Salem, S.: Selecting discriminating terms for bug assignment: A formal analysis. In: Proceedings of the 7th International Conference on Predictive Models in Software Engineering. pp. 1–7. New York, USA (2011)

3. Alpaydın, E.: Combined 5× 2 cv f test for comparing supervised classification learning algorithms. Neural Computation 11(8), 1885–1892 (1999)

4. Anvik, J., Hiew, L., Murphy, G.C.: Who should fix this bug? In: Proceedings of the 28th International Conference on Software Engineering. pp. 361–370. New York, USA (2006)

5. Banitaan, S., Alenezi, M.: Tram: An approach for assigning bug reports using their metadata. In: Third International Conference on Communications and Information Technology (ICCIT). pp. 215–219. Beirut, Lebanon (2013)

6. Bertram, D., Voida, A., Greenberg, S., Walker, R.: Communication, collaboration, and bugs: The social nature of issue tracking in small, collocated teams. In: Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work. pp. 291–300. New York, USA (2010)

8. Bhattacharya, P., Neamtiu, I., Shelton, C.R.: Automated, highly-accurate, bug assignment using machine learning and tossing graphs. Journal of Systems and Software 85(10), 2275–2292 (2012)

9. Brownlee, J.: How to choose a feature selection method for machine learning. 2019, https://machinelearningmastery.com/feature-selection-withreal-and-categorical-data/, accessed on September 17, 2022

10. Bulat, O.: Zeyrek: A turkish morphological analyzer and disambiguator. 2020, https://github.com/obulat/zeyrek, accessed on June 10, 2023

11. Clark, K., Luong, M.T., Le, Q.V., Manning, C.D.: Electra: Pre-training text encoders as discriminators rather than generators, arXiv:2003.10555 [cs], 2020

12. Dash, M., Liu, H.: Feature selection for classification. Intelligent Data Analysis 1(1-4), 131–156 (1997)

13. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv:1810.04805 [cs], 2018

14. Dietterich, T.G.: Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation 10(7), 1895–1923 (1998)

15. Feng, L., Senapati, J., Liu, B.: Tadaa: Real-time ticket assignment deep learning auto advisor for customer support, help desk, and issue ticketing systems, arXiv:2207.11187 [cs], 2022

16. Florea, A.C., Anvik, J., Andonie, R.: Parallel implementation of a bug report assignment recommender using deep learning. In: International Conference on Artificial Neural Networks. pp. 64–71. Cham, Switzerland (2017)

17. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)

18. Goyal, A., Sardana, N.: Machine learning or information retrieval techniques for bug triaging: Which is better? e-Informatica Software Engineering Journal 11(1), 117–141 (2017)

19. Gupta, S., Gupta, S.: Bug reports and deep learning models. International Journal of Computer Science and Mobile Computing 10(12), 21–26 (2021)

20. Guven, Z.A.: The comparison of language models with a novel text filtering approach for turkish sentiment analysis. ACM Transactions on Asian and LowResource Language Information Processing 22(2), 1–16 (2022)

21. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: An update. ACM SIGKDD Explorations Newsletter 11(1), 10–18 (2009)

22. Herbold, S., Trautsch, A., Trautsch, F.: On the feasibility of automated prediction of bug and non-bug issues. Empirical Software Engineering 25(6), 5333–5369 (2020)

23. Hern´andez-Gonz´alez, J., Rodriguez, D., Inza, I., Harrison, R., Lozano, J.A.: Learning to classify software defects from crowds: A novel approach. Applied Soft Computing 62(1), 579–591 (2018)

24. Hosseini, H., Nguyen, R., Godfrey, M.W.: A market-based bug allocation mechanism using predictive bug lifetimes. In: 16th European Conference on Software Maintenance and Reengineering. pp. 149–158. Szeged, Hungary (2012)

25. Jonsson, L., Borg, M., Broman, D., Sandahl, K., Eldh, S., Runeson, P.: Automated bug assignment: Ensemble-based machine learning in large scale industrial contexts. Empirical Software Engineering 21(4), 1533–1578 (2016)

26. Karim, M.R., Ruhe, G., Rahman, M.M., Garousi, V., Zimmermann, T.: An empirical investigation of single-objective and multiobjective evolutionary algorithms for developer’s assignment to bugs. Journal of Software: Evolution and Process 28(12), 1025–1060 (2016)

27. K¨oksal, O., Tekinerdogan, B.: Automated classification of unstructured bilingual ¨ software bug reports: An industrial case study research. Applied Sciences 12(1), 338 (2021)

28. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach, arXiv:1907.11692 [cs], 2019

29. Loria, S.: Textblob. 2020, https://textblob.readthedocs.io (accessed on June 10, 2023)

30. Mahajan, G., Chaudhary, N.: Bug classifying and assigning system (bcas): An automated framework to classify and assign bugs. In: Smart Systems: Innovations in Computing: Proceedings of SSIC. pp. 537–547. Jaipur, India (2022)

31. Mani, S., Sankaran, A., Aralikatte, R.: Deeptriage: Exploring the effectiveness of deep learning for bug triaging. In: Proceedings of the ACM India Joint International Conference on Data Science and Management of Data. pp. 171–179. New York, USA (2019)

32. McCallum, A., Nigam, K.: Bow: A toolkit for statistical language modeling, text retrieval, classification, and clustering. Journal of Machine Learning Research 2(3), 557–559 (2000)

33. Merten, T., Falis, M., H¨ubner, P., Quirchmayr, T., B¨ursner, S., Paech, B.: Software feature request detection in issue tracking systems. In: IEEE 24th International Requirements Engineering Conference (RE). pp. 166–175. Beijing, China (2016)

34. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space, arXiv:1301.3781 [cs], 2013

35. Odegua, R.: An empirical study of ensemble techniques (bagging, boosting, and stacking). In: Proceedings of the Deep Learning IndabaX. pp. 25–31. Nairobi, Kenya (2019)

36. Oliveira, P., Andrade, R.M., Barreto, I., Nogueira, T.P., Bueno, L.M.: Issue auto assignment in software projects with machine learning techniques. In: IEEE/ACM 8th International Workshop on Software Engineering Research and Industrial Practice (SER&IP). pp. 65–72. Madrid, Spain (2021)

37. Panda, R.R., Nagwani, N.K.: Topic modeling and intuitionistic fuzzy set-based approach for efficient software bug triaging. Knowledge and Information Systems 64(11), 3081–3111 (2022)

38. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in python. Journal of Machine Learning Research 12(85), 2825–2830 (2011)

39. Peng, X., Zhou, P., Liu, J., Chen, X.: Improving bug triage with relevant search. In: International Conference on Software Engineering and Knowledge Engineering. pp. 123–128. Pittsburgh, USA (2017)

40. Rehurek, R., Sojka, P.: Gensim - python framework for vector space modelling. Masaryk University NLP Centre 3(2), 2 (2011)

41. Robertson, S.: Understanding inverse document frequency: On theoretical arguments for idf. Journal of Documentation 60(5), 503–520 (2004)

42. Sajedi-Badashian, A., Stroulia, E.: Vocabulary and time based bug-assignment: A recommender system for open-source projects. Software: Practice and Experience 50(8), 1539–1564 (2020)

43. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: Smaller, faster, cheaper, and lighter, arXiv:1910.01108 [cs], 2019

44. Schweter, S.: BERTurk - BERT Models for Turkish. 2020, https://github.\com/ stefan-it/turkish-bert (accessed on September 17, 2022)

45. Sebastiani, F.: Machine learning in automated text categorization. ACM Computing Surveys (CSUR) 34(1), 1–47 (2002)

46. Sharma, M., Tandon, A., Kumari, M., Singh, V.: Reduction of redundant rules in association rule mining-based bug assignment. International Journal of Reliability, Quality and Safety Engineering 24(06), 1740005 (2017)

47. Shokripour, R., Anvik, J., Kasirun, Z.M., Zamani, S.: A time-based approach to automatic bug report assignment. Journal of Systems and Software 102, 109–122 (2015)

48. Singh, V., Chaturvedi, K.K.: Bug tracking and reliability assessment system (btras). International Journal of Software Engineering and Its Applications 5(4), 1–14 (2011)

49. Stellman, A., Greene, J.: Applied Software Project Management. O’Reilly Media, California (2005)

50. Su, Y., Xing, Z., Peng, X., Xia, X., Wang, C., Xu, X., Zhu, L.: Reducing bug triaging confusion by learning from mistakes with a bug tossing knowledge graph. In: 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). pp. 123–128. Melbourne, Australia (2017)

51. Sureka, A.: Learning to classify bug reports into components. In: Objects, Models, Components, Patterns: 50th International Conference. pp. 288–303. Prague, Czech Republic (2012)

52. Tunali, V.: Improved prioritization of software development demands in turkish with deep learning-based nlp. IEEE Access 10, 40249–40263 (2022)

53. Weiss, C., Premraj, R., Zimmermann, T., Zeller, A.: How long will it take to fix this bug? In: Fourth International Workshop on Mining Software Repositories (MSR’07: ICSE Workshops). p. 1. Minneapolis, USA (2007)

54. Wohlin, C., Runeson, P., H¨ost, M., Ohlsson, M.C., Regnell, B., Wessl´en, A.: Experimentation in Software Engineering. Springer, Berlin (2012)

55. Xia, X., Lo, D., Ding, Y., Al-Kofahi, J.M., Nguyen, T.N., Wang, X.: Improving automated bug triaging with specialized topic model. IEEE Transactions on Software Engineering 43(3), 272–297 (2016)

56. Xuan, J., Jiang, H., Ren, Z., Yan, J., Luo, Z.: Automatic bug triage using semisupervised text classification. In: International Conference on Software Engineering and Knowledge Engineering. pp. 209–214. San Francisco, USA (2010)

57. Yang, G., Zhang, T., Lee, B.: Utilizing a multi-developer network-based developer recommendation algorithm to fix bugs effectively. In: Proceedings of the 29th Annual ACM Symposium on Applied Computing. pp. 1134–1139. Gyeongju, Republic of Korea (2014)

58. Yang, X., Lo, D., Xia, X., Bao, L., Sun, J.: Combining word embedding with information retrieval to recommend similar bug reports. In: IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). pp. 127–137. Ottawa, Canada (2016)

59. Yilmaz, O.: Turkish pos tagger. 2015, https://github.com/onuryilmaz/ turkish-\pos-tagger, accessed on June 10, 2023

60. Zhang, H., Gong, L., Versteeg, S.: Predicting bug-fixing time: An empirical study of commercial software projects. In: 35th International Conference on Software Engineering (ICSE). pp. 1042–1051. San Francisco, USA (2013)

61. Zhou, Z.H.: Ensemble Methods: Foundations and Algorithms. CRC Press, Florida (2012)