API Documentation

       
                                                                                                        
                                                                                                                                                                                                                            {"pages_text_array":["Introduction to Big DataLearning ObjectivesAt the end of this text, you should present the   following learnings: Define big data.Discuss the Vs of big data and implications.Point out the   types of data related to big data.IntroductionSince the beginning, man    has stored data for   himself and for others, through drawings on the rocks and rock art. This record was made with   the aim of making some decision or enabling access to knowledge. As societies became more   complex, the volume of data storage This led to the    construction of libraries and the later   invention of printing by Johannes Gutenberg around 1450. The abacus itself, a mechanical   instrument of Chinese origin created in the 5th century BC, stored information about numbers   and helped with computing. Later,    the emergence of the internet for information exchange,   during World War II and the Cold War (1945\u01521991), made it even more necessary data   storage for further analysis. Over time, various ways of storing this information were   developed: mainframes, floppy    disks, tapes, hard drives, NAS (Network  - Attached Storage),   cluster environment, pen drives, CDs, DVDs. In modern society, data began to be produced   from different sources, whether in social networks (photos, videos, messages), in online   purchases, in deli very applications, in distance education courses, in transactions with   currencies and digital banks. In addition, there was the replacement of roles, such as physical   agenda, medical records, request for exams, for the digital context.           In this sense, companies realized the value of storing and processing strategic data. Thus, the   new power race became clear: data started to be seen as the new oil. As a result, we can   observe a gradual growth in the production and storage of data througho  ut history, until we   reach the context of big data. In this chapter, you will study the concept and characteristics   inherent to big data. the main types of data that are related to context.1 The data society and   what defines big dataModern culture started   to produce and store more data. With a   computer or a smartphone in our hands, we now have access to a greater volume of   information. Thus, the massive growth of sending photos, videos, audio and text messages   made the social relationship become digital. It    was in this scenario that the concept of big data   emerged. According to Mauro, Greco and Grimaldi (2015, online document, our translation),   big data is defined as follows: \ufb01Big Data represents information assets characterized by high   volume, speed and var  iety, which require specific technologies and analysis methods to be   transformed in value [...]\ufb02.From the growth of hundreds of Terabytes of data, the context of   big data began to be systematized.The definition of this term is based on five principles: spe  ed,   volume, variety, veracity and value .You will see that such principles always go together in this   context. Thus, big data is a broad term that deals with several areas and composes the various   related studies. In the academic area, departments were cre  ated focused on engineering and   data science. , in order to compose the sets of knowledge and studies that the area demanded.   Soon, several professions related to this area also emerged. The data engineer deals with   acquisition, storage and disposal strate  gies. level of data. The data scientist and the machine   learning engineer (in English, machine learning) make up the context of exploratory analysis,   pattern recognition and predictive analysis, as well as other related contexts. similar to   software engine ering DevOps, but focused on the context of the data.Introduction to big data2         ","The exponential production of data with the internet of thingsThe internet of things has   emerged with remarkable potential, causing the context of connected devices to exponen  tially   increase agricultural production produces voluminous data every second, with monitoring in   the chicken coop, monitoring the temperature and ambient humidity, among others. As a   result, a large volume of data is produced. The production of refrigerat  ors, air conditioners,   fans, electric pans and other connected devices made daily life permeated by the internet of   things .Thus, with so much data produced, it is necessary to organize a storage and processing   structure for decision making. The term \ufb01inte  rnet of things\ufb02 refers to the interconnection of   intelligent devices, which produce, consume and transmit data. make up the context, in   addition to several boards and embedded systems.The production of data by peopleThe   biggest producers of given   Away       s in  the world are human beings themselves. Previously, each could only create small notes for   themselves or within a small group. Now, we have a massive online file sharing environment at   our disposal. As we walk, we produce data through of our GPS positions,   which are transmitted   in real time via applications. Our speech produces data that is analyzed by virtual assistants. If   we are hospitalized, our breathing will produce data, through sensors, for the medical record.   the use of social networks is increasing  , generating immense amounts of data. The sending of   daily e - mails with advertisements and for closing deals, the allocation of photos of travel in the   cloud and several other situations of our daily life generate data, in tremendous volume and   speed.So, i n the age of big data, to live is to produce data.3Introduction to big data           The production of public data by governmentsGovernments also produce a tremendous range   of data, on the most diverse fronts: health, infrastructure, transport, education, tourism  ,   economy, bids, contracts, among others. Federal Government website and are commonly   consumed by entities that In addition, the market seeks to carry out, from this data, various   predictive analyses. On the other hand, governments also use each other's da  ta, 2 The Vs of   big data and its impact on technologies and society Big data has changed the way companies   see their data. Currently, each piece of information about their own business and customer   has become crucial in decision making. In the academic con  text, more and more processing   and data analysis. In this sense, the characteristics of big data and its five Vs showed the   systematization of the context, offering a vision of how studies and technological solutions   should be. those proposed for the area.    See below for more information on each of these   aspects.Volume The reference to the size of the data produced and the need to store it   encompasses the volume of big data. Currently, we are not talking about Terabytes, but about   Zettabytes or of Brotonbyte  s.Speed  The speed in data production can be seen, for example,   from the perspective of social networks, where we have millions of messages exchanged per   minute.Imagine that a million people sent 10 messages only in the morning, that is, in the first   six  hours of your day. In that case, we would already have Introduction to big data4         ","10 million pieces of data to be stored. The reality, however, is much greater. The production of   data is fast, whether in monitoring, through sensors, or in the data that pe  ople themselves   produce. VarietyThe multiplicity of file types within of big data is, in fact, a punctual   characteristic. When we started to produce, mostly, digital data, we transformed physical tasks   into online data. This data can be agendas, purchase o  rders and deliveries, sending text   messages , audio, video and image. This variety can be composed and stored, for example, in   the HDFS file structure of Apache Hadoop, and managed by its various services, such as Hive,   Hbase, Spark, among others. Veracity  The composition of the veracity of the data in big data is   a characteristic part of data quality and continuous improvement. We cannot use data that do   not represent the problem or that have a bias. In this context, data science deals with cleaning   and org anizing the data, in order to increase the framework for the context of data quality   comes from The Dama guide to the data management body of knowledge that help in data   governance.ValorThe first step that occurred in big data was the need to store the dat  a, for   only later see what to do with them. This was because it was realized that, with the rise of   predictive analytics, having a lot of data about a given context could be invaluable. .The use of   data for companies was already used in business intelligen  ce, which became known for the   theories of the data warehouse and the respective specificities, with techniques to create data   structures and enter the dashboards. However, predictive analysis has gained immense   importance , since everyone wants to predict    the future based on several variables in a   context.5Introduction to big data           Other VsAccording to Taleb, Serhani and Dssouli (2019), there are still other Vs involved. Some   of them are variability, which consists of the constant change of security issue  s. Data ingestion   and storage in Apache HadoopData can take different forms. structured as spreadsheets, in   ERP systems, they can be semi  - structured or unstructured, like data from social networks, or   they can come from a network of wireless sensors that p  roduce information such as   temperature, humidity or pressure (Figure 1).Figure 1.The various data types of   "],"pdf_complete_text":"Introduction to Big DataLearning ObjectivesAt the end of this text, you should present the   following learnings: Define big data.Discuss the Vs of big data and implications.Point out the   types of data related to big data.IntroductionSince the beginning, man    has stored data for   himself and for...