Practical approaches to big data privacy over time

Key Points Governments and businesses are increasingly collecting, analysing, and sharing detailed information about individuals over long periods of time. Vast quantities of data from new sources and novel methods for large-scale data analysis promise to yield deeper understanding of human characteristics, behaviour, and relationships and advance the state of science, public policy, and innovation. The collection and use of fine-grained personal data over time, at the same time, is associated with significant risks to individuals, groups, and society at large. This article examines a range of long-term research studies in order to identify the characteristics that drive their unique sets of risks and benefits and the practices established to protect research data subjects from long-term privacy risks. We find that many big data activities in government and industry settings have characteristics and risks similar to those of long-term research studies, but are subject to less oversight and control. We argue that the risks posed by big data over time can best be understood as a function of temporal factors comprising age, period, and frequency and non-temporal factors such as population diversity, sample size, dimensionality, and intended analytic use. Increasing complexity in any of these factors, individually or in combination, creates heightened risks that are not readily addressable through traditional de-identification and process controls. We provide practical recommendations for big data privacy controls based on the risk factors present in a specific case and informed by recent insights from the state of the art and practice. Commercial and government actors are collecting, storing, analysing, and sharing increasingly greater quantities of personal information about individuals over progressively longer periods of time. Advances in technology, such as the proliferation of Global Positioning System (GPS) receivers and highly-accurate sensors embedded in consumer devices, are leading to new sources of data that offer data at more frequent intervals and at finer levels of detail. New methods of data storage such as cloud storage are more efficient and less costly than previous technologies and are contributing to large amounts of data being retained for longer periods.1 Powerful analytical capabilities, including emerging machine learning techniques, are enabling the mining of large-scale datasets to infer new insights about human characteristics and behaviours and driving demand for large-scale datasets. These developments make it possible to measure human activity at more frequent intervals, collect and store data describing longer periods of activity, analyse data long after the data were collected, and draw inferences from a large number of individual attributes. Moreover, analytic uses and samples sizes are expanding with emerging big data techniques. Taken together, these factors are leading organizations to collect, store, and use more data about individuals than ever before, putting pressure on traditional measures for protecting privacy. Commercial and government data accumulating over time make it possible to paint an incredibly detailed portrait of an individual’s life, making such data highly valuable not only to the organizations collecting the data but to secondary users as well. These data are increasingly being made available to researchers, policymakers, and entrepreneurs, in support of rapid advances in public policy, and These data increasingly long periods of time and at increasingly frequent intervals, enabling insights that can only be from fine-grained datasets over time. Commercial big data are and to provide and to and to and make or and collect, store, and analyse large quantities of data about and detailed personal information is to provide and to and Governments are with collecting increasingly detailed information in order to the of their from and to and and to their and of big data promise to yield significant in the and government long-term data collection research in the and of the studies, the the of risk factors for and many advances in studies have on understanding in such as and The of large-scale data from and government and big data analysis techniques, such as emerging machine learning to the in including of science, in The and use of large quantities of personal data for periods of time is the subject of recent and as as the to be and a These developments are to have for the of privacy in the big data and the of their is to be long-term data activities have for at a in research settings in that are similar to and government data human subjects research is associated with significant to the collection of a large of personal information about These are not understood on a as risks in over the of a long-term the that are of in an have and and are in the of and the human subjects The that a of information from a an individual to such as a of of in the or and or or and government settings a number of privacy policy, and these settings is the to protect individuals and from to and sharing information about research is to draw inferences about the population being and collecting and protecting information about the same that are in and government and use methods from such as of behaviour, and to data about their and through on and individual in big data and collect data in from to and to readily behaviour, including and a research that large of information about individuals collect personal information about individuals such as their and relationships to individuals, and personal information is in the through and such as those by or such information can be readily from data such as or these settings similar and of it have the case that data on consumer behaviours and their data are being to make inferences about a range of and factors and with an individual’s use of a or and make inferences that research for the of but for of consumer that are in and risk of personal information in the public the of similar for such as long-term data activities have the to individuals, groups, and society at large to greater risks that personal information be or in that in the of research in long-term data activities drive heightened privacy risks. research studies are associated with large of about data and these data are and such as or or or The of the data enabling research that be the same time, fine-grained about individuals, such as their activity, or are to be associated with such as a and and and activities with are at the of an individual subject and to in individual’s and characteristics over an of such detailed information individuals to to their and personal risk of of and risks of and and or in in the a of information privacy risks in research that the data characteristics to time is as the of time that the data collection and be after as in the case of a that an based on the or data be after including government that are from for many to the of the subjects are data make at a in time, such as a in collection over and such as a long-term research or a or the measures on the same of data collection and that data such as and the of the studies collect data from a or as in the factors are the privacy risks associated with big data in and government these factors are not with time, are increasingly associated with data or the number of for data of data datasets of of that are by data and data that only or the of analysis the data are intended to studies are to support or inferences about and government have such as making inferences about individuals or in such as to size, to the number of in the of data collection in and government settings have sample sizes than traditional research studies, and such data sources a of the population from the sample is characteristics, to the and of the population from in the data are research studies are from or big data in settings or to of these with in long-term data and The state of the for privacy and to these developments in The risks that can be for understanding new be in the government and research have in order to protect privacy personal practices the in and government are and and de-identification techniques. These are or on use of data that have to a of or This on a of and use of privacy from practices in research and have and a large collection of for the long-term risks of collecting and personal data about human the risks and benefits of their research activities to individual subjects and society at and a of to protect subjects the collection of data to that is uses of the data to those at the of the and the of personal a range of privacy including and informed over time, to learning about information specific to individuals, data use and controls to data and use. and have and over time to provide privacy for research The to research by of the that have to the or at an that to with the be by state protecting the of human research or by the research data of their and studies human subjects their for by an and and that subjects be informed of the and of the the of personal information to be collected, the research and data to be and to be to and any risks and benefits to in the informed and risks and benefits to research and subjects are informed of risks and the benefits the risks. a long-term research with on an or more frequent studies to the behaviour, and of individuals, data a of information by large number of and state protect data the on the of and uses to and or the and the and and the and collecting data on of are to privacy in with such as the of and the and range of and controls are at of a long-term research from research subjects is in the of a that the of the with a subject and uses of personal information are to the research in the the of information or is the the or the of the population being the new from as new research the collecting and from samples and to new in the factors data are to support analysis methods and research not at the time of the a detailed of for subject new data collection or use activity, it is by the on provide research including sharing of data with researchers, and sharing of data with to use for specific of such as and and research and use data and use have by data and data the and of the the on and the data and the data and to be the of the and and oversight and of the of such are are in the long-term research is significant in and for the individuals have to personal data over the of a practices for data are by of such as government and and such specific for data at collection or such as or of personal to and data of data collection in a the of to the personal sharing long-term research are in The of the data and the to draw to sources the of data in be to provide public to datasets and datasets to use a of techniques, such as and to a These but is a that such provide only over the long are to specific of such as sources of data to of are to in the or of The data privacy and is more for long-term for risk have with data at in time or to in time, in and are for data privacy that are for datasets in privacy or a greater in data to studies collect data at the of an individual for the of in individual behaviour, data that have or are not for not by the of the are to the of data in that and inferences and make of analysis Moreover, privacy made to a are not to the a data as an data leading to and on their such as data and are less to protect the of long-term new are being for in the of a that as a or in to from are that to the that is to studies, in datasets are and over time. a similar the is to studies, research to that information only a number of of the these are in such be to of or be to risks for inferences that are with the for to the of risks associated with of data over the of a long-term controls. or with of use or data use are to and sharing data with and secondary to to the and to of researchers, such as or a of be to in or to with a data for large be to data only through or data and uses of data in a and for long-term data activities in and government settings from those in the research and privacy risks at the time a data collection but in not in and with long-term risks in on such as and or than from the range of privacy that are available and of privacy controls the information privacy controls for similar data research and government by and including oversight by an to and a that to to data privacy and data are in is risk by and by research in of privacy risks and for long-term and of including and and controls. on a of privacy controls such as and and analysis and controls are not to of big are increasingly of and with and to big data and government and and information privacy such as and a significant in protecting research data but to only a of and government data organizations to as a best of the by such data activities be by that to researchers, such as the the and of the such as the of the and and the of as as state activities of government privacy such as data but and a of such as and are established by the and by such as the of and These and to leading to in practice. best such as the and the are on data such as and than privacy can be about individuals to the data actors to the information for these are at a than practices through detailed The of these is in a industry with a long and significant large quantities of information about these are in many are not with specific and and controls. and for data are not as as those for human subjects The of and oversight by an or similar in less on subjects of risks and data collection and and controls to long-term risks. an understanding of the to their personal data are collected, and by Commercial data are with data from sources and to research data not be with data in in the research and by an and and to by the individual to the informed process in practices are through privacy or of that is not at by is in and frequent with users provide to is a that a on and is not or privacy and the of such are or as to not those state and data sharing and in the of data The to through a to the of that an as a of a many the are to in the is an in the risk that a that the are or The be as of or such as a that and it be to that a a are to the and of information about their data and and privacy are are in and to privacy and measures in a are by and by to privacy but activities less of long-term risks. data for to be the for information and data to the the to data for a specific such as or being made available for use by the public for any These in large by and state including the of in to public that the are significant to or that of a of such as and by of such as and in large to the of detailed are in an and practices are not informed of the specific of their personal of the associated benefits and to and government not for long-term and of risks. the to and for data or to their privacy and from users to or use a of data that not in their privacy or use their data in a new or for a than at large data are to these practices in of the and of studies in the research The associated with big data have not and to the same that the associated with long-term research studies have long by and and to data that are with are increasingly and data and privacy number of large are to established an process for research based on have made to individuals to of data collection and have made of their data and by data have made only a of the are to their collection and use of personal in the of privacy for as by and of privacy privacy in their data collection or in the use of personal data is such as in to and research of by their are of personal data and uses that less on the of personal information of have for their big data such not the of and by a is established an research process by the in the and in the for on privacy and research by with in the of by a of and in and The of the research to the and society at to and and for are these the from the with to or and to the research is with their personal information is collected, and research and recommendations by and is is by measures the on in the The and personal data from a number of sources including public and and protect the data it with a number of and industry including those in and The to in privacy with a on use of a analysis to a with of and of or or on these a and are it is in an in practice. the of a data use made by the The a of including personal for a number of such as and the to the use of such be as of the to the on these for of the individuals in the with the in personal the that were and not be made available as a to This to in be based on by a number of and their about The in that not to in such a the have made a the use of the at The on the of an or a of individuals, use of data is to to practices in the of a have the of similar of highly information about data have made to of of of and of individuals from including is it is of the of data privacy a large number of and government data activities privacy risk in and to the of traditional to privacy. and are new and for data privacy that can traditional and emerging The of time on privacy risk is and not are embedded in a of privacy and the and can the that risk privacy risk as a function of and and are in of information is risk is not an function of these and be as factors, and their are not in the as individuals be more to of describing for data in and from the of together, at characteristics to time as that data privacy age, period, and the these temporal characteristics and the of privacy risk and as Key risk for big data over time and their on privacy risk information is to the risk of as characteristics and the and of data have over time. an individual not making less with time. is a as characteristics such as not over time. The of sources of information that be to infer information about individuals in a is and are being and made available the to and enabling uses from the at the time the data were These factors traditional to privacy such as the of in large on sources of information that can be to information in a are over the risk of data the of data in and it can be the time data collection and use the for the data to uses that not be at the time of collection from data use. New analytical such as machine learning and can new uses of information for are being to the of measure risk factors, and infer The more time that collection and the greater the that to with privacy and and risks to data in with the to be in the based on such as and of the and the of the data is of the factors on such as an individual’s as a public over time, or the of a for a after many have the of the data is less such as an individual a for a or and is for an with or data individuals in the data less to in the be as less and less to for the of from a of a at a time the subject is a or a recent such a This is in the for protecting in information to the of time as in the risk the of the data is the subjects be and by many of the of personal information on the or on to or information from is for from the of after the risk is to be to public such as the only to data about and research studies are to data after a of time over an in as analysis of over time characteristics to behaviour, and the have collecting data from and their and the data information about an individual’s of risk factors for or of and and is the of on the collection of data and such as and and research to and and and the factors controls on to and use of detailed data from including data have in large to about risks from about individuals over a long are risks to and periods of data collection are with greater of the as be at as large as the period, and privacy human at temporal the of of collection and frequency of and enabling at frequent intervals can or about and use sensors to and to an individual’s and a research in the activity levels of in their by the number of and by activity data with as as data on an individual’s or being to identify many and government big data collection to more frequent than those in the research and sensors embedded in a can fine-grained by and the data to the cloud for long-term storage and in frequency in data analysis and to uses of personal information an of data and of of and of heightened for privacy frequent analysis of data about the same individuals the risk of learning information specific to the individuals in the data not a privacy by is a for and is associated with from analysis and including data to privacy in such as are to protect such risks and the to collect and analyse data at greater frequency benefits for research and it can to greater to individuals, and to a greater number of individuals, the data be are associated with temporal frequency data collection with long periods of data the from data to the of individual in a can with the of the be associated with a greater of sources of information that be to individual of the and the state of the art for privacy are of the of data the of privacy risk and non-temporal characteristics of big including the and of the population being the of the the of the and the analytic uses by the characteristics of big data that drive privacy risks and their on of privacy risk or the number of for data is a with privacy from a data is and of on the number of for data subject in order to analyse the risks associated with these factors This it possible to analyse data heightened risks to the of information about the same the to for individuals, and the of measures of the same to of that are over time. The of measures can unique or of that can be to a individual with a in the were not in a of sets of were to be of an individual’s and personal of research data from based on the large number of by of by users to to the number of from making many of the in the unique and by with risks of an analysis of public information by on the can be to personal characteristics, such as an individual’s of about the use of personal information to in that individuals and are on to those in the research have an to as many data as and draw at the individual in order to more about individuals in their and data about the same individuals from many including from government and data from The are to including and and to more data of data on individuals, including about of information about and these are to for data are to a of or The of big data be greater than it as data such as and are subject to risks from are a function not only of the but the of the of be associated with have embedded such as and and be to through such as those by of and data a range of for of have based on and can be it be possible to from of on or to the of a based on be highly enabling that are only possible at levels of for can the or an individual of privacy risk in big data is the for analytic including uses that be as New big data sources and analytical techniques, including a of are enabling of individuals in such as to to by to in that or of a based on the to the is by learning about individual from of the is to or based on personal information but such are not as from to data and to individuals based on their attributes. at the of the is in the is to make inferences about the of a specific and the of that be those that from population through or to privacy such as individual and to of techniques, as as based on privacy such as are to of population and not protect learning about that be to These are not the is to make inferences about or with to in sample are associated with greater risks. a sample to a of can be more that any individual is in the samples to be are to of be to the greater for such as and Commercial not by these collect and analyse personal information about such the of the sample is large with to the it can support uses such as more than risks of for society and a population in a of data the range of that are to at of the of not a large for a only individuals, it for a of a population that individuals order to of similar risks be This for privacy and in data more and best practices established by the research can be for privacy with to the long-term collection and use of personal data by and government analysis of the and intended uses associated with a of data can be to the of sets of privacy and in the research a of the and the of and controls at the collection and of the information on a of from the range of and that are available to data data be to from the of available and at of the information from to and and the recommendations of sets of privacy and controls. This is from a to in than on a such as de-identification or and government actors the of of a range of is a that de-identification is not as a New and for and privacy and and and are being and are to be as of data data sharing can and a data be to provide public to data after have the data for users to that the including or be to an to a use of the data be by the of a data use in only through a data data can be to the and associated with a of and the uses by in the characteristics of long-term data such as the frequency of the sets of controls. in of the big data to in and as in the a big data characteristics recommendations of privacy and controls. a to collect data from the than on the The to offer users to of their The to collect data over a longer of time and to on the over The a to users to in their the to the the of a to collect data on a from as the in the than on users to their the This a from data collection to frequency data is to the of the data collected, as data on a of or that can be to the of an from to collection is not to more about individuals in in a on the of The large in and in the of to the for controls. frequency from to of the data collection on an is to an individual’s of and and frequency to samples the collection