Research on IT Architecture of Heterogeneous Big Data

Yun Liu; Qi Wang; Hai-Qiang Chen

doi:10.6180/jase.2015.18.2.05

Research on IT Architecture of Heterogeneous Big Data

Computer Science and Information Engineering

Yun Liu This email address is being protected from spambots. You need JavaScript enabled to view it.¹, Qi Wang¹ and Hai-Qiang Chen¹

¹School of Communication and Information Engineering, Key Laboratory of Communication and Information Systems, Beijing Municipal Commission of Education, Beijing JiaoTong University, Beijing 100044, P.R. China

Received: May 14, 2014
Accepted: May 20, 2015
Publication Date: June 1, 2015

Download Citation: ||https://doi.org/10.6180/jase.2015.18.2.05

ABSTRACT

The amount of data has grown exponentially in industry and internet, and we are facing a significant problem of information explosion. The challenge is not only to store and manage the vast volume of data (“big data”), but also to analyze and extract meaningful value from it. This paper analyzed the characters of the future network and the features of data generated by it. Furthermore, we studied key issues and challenges of the multiple heterogeneous data IT infrastructure. The main focuses of this paper are on three aspects of distributed storage, cloud computing, and data fusion. There are several techniques to address these issues. At last, we proposed an architectural solution and key technology program which can be adapted to developing more application functionality and meet future demand for Internet services and new business for big data processing requirements.

Keywords: Cloud Computing, Hadoop, Data Mining, Distributed Storage

REFERENCES

[1] Chang, F., Dean, J., Ghemawat, S., et al., “Bigtable: A Distributed Storage System for Structured Data,” ACM Transactions on Computer Systems, Vol. 26, Issue 2, Article No. 4 (2008). doi: 10.1145/1365815. 1365816
[2] Ghemawat, S., Gobioff, H. and Leung, S. T., “The Google File System,” ACM SIGOPS Operating Systems Review. ACM, Vol. 37, Issue 5, pp. 2943 (2003). doi: 10.1145/1165389.945450
[3] Dean, J. and Ghemawat, S., “MapReduce: Simplified Data Processing on Large Clusters,” Communications of the ACM, Vol. 51, Issue 1, pp. 107113 (2008). doi: 10.1145/1629175.1629198
[4] White, T., Hadoop: The Definitive Guide. 1st, O’Reilly Media (2009).
[5] Information on http://research.yahoo.com/files/ycsb. pdf.
[6] Information on https://www.google.com/patents/US2 0030051097.
[7] Hall, D. L. and Llinas, J., “An Introduction to Multisensor Fusion,” Proceedings of the IEEE, Vol. 85, Issue 1, pp. 623 (1997). doi: 10.1109/5.554205
[8] Walts, E. L., Data Fusion for C3I: a Tutorial, in: Command, Control, Communications Intelligence (C3I) Handbook, EW Communications Inc., Palo Alto, CA, pp. 217226 (1986).
[9] White, F. E., Data Fusion Lexicon, Joint Directors of Laboratories, Technical Panel for C3, Data Fusion Sub-Panel, Naval Ocean Systems Center, San Diego (1991).
[10] Steinberg, A. N., Bowman, C. L. and White, F. E., “Revisions to the JDL Data Fusion Model,” in: Proc. of the SPIE Conference on Sensor Fusion: Architectures, Al - gorithms, and Applications III, pp. 430441 (1999). doi: 10.1117/12.341367
[11] Llinas, J., Bowman, C., Rogova, G., Steinberg, A., Waltz, E. and White, F. E., “Revisiting the JDL Data Fusion Model II,” in: Proc. of the International Conference on Information Fusion, pp. 12181230 (2004). doi: 10.1109/ICIF.2005.1591959
[12] Dasarathy, B. V., Decision Fusion, IEEE Computer Society Press, Los Alamitos CA (1994).
[13] Kokar, M. M., Tomasik, J. A. and Weyman, J., “Formalizing Classes of Information Fusion Systems,” Information Fusion, Vol. 5, No. 3, pp. 189202 (2004). doi: 10.1016/j.inffus.2003.11.001
[14] Champion, M., Ferris, C., Newcomer, E., et al., Web Services Architecture, W3C Working Draft, (2002-11- 14). http://www.w3.org/TR/ws-arch.html.
[15] Huang, S. G., Fan, Y. S., Zhao, D., et al., “Web Service Based Enterprise Application Integration,” Computer Integrated Manufacturing Systems, Vol. 9, No. 10, pp. 864867 (2003).
[16] Hung, M.-H., Cheng, F.-T. and Yeh, S.-C., “Development of a Web-Services-based E-diagnostics Framework for Semiconductor Manufacturing Industry,” IEEE Transactions on Semiconductor Manufacturing, Vol. 18, No. 1, pp. 122135 (2005). doi: 10.1109/TSM. 2004.836664
[17] Otal, M. and Jelinekl, I., “The Method of Unified Internet-based Communication for Manufacturing Companies,” Lecture Notes in Computer Science, Springer, Vol. 3190, pp. 133140 (2004). doi: 10.1007/978-3- 540-30103-5_15