Big data offers us big potential, for both benefits and risks. Although it has great appeal, we need to strike a healthy balance – to achieve maximum benefit for an acceptable level of risk. Security, privacy, and compliance considerations and solutions should be integral to any big data project.
The value of your data and insight grows as your volume does. Big data aggregates significant volumes of information, which leads to an increased interest from attackers and impact if compromised. Aggregation can be due to the accumulation of data or by associations the data enables.
Infrastructure, systems, applications, databases, processes, transactions and audit logs must all be properly secured; limiting access and rights to only those allowed. You must maintain the confidentiality, integrity, and availability of your critical data, against both external and internal threats.
Security needs to be supported by an ability to audit use and deal with misuse, including incident investigation, digital forensics, disciplinary action, and communications plan. You need to be prepared for the worst.
The associations that big data helps us make can also lead to more personal intrusion than might be understood or accepted. Analysing medical history, browsing and buying patterns, communications metadata, and other data sets, can provide quite a complete view of private lives.
When developing a system that will process large volumes of records, especially if they are sensitive, you should also involve your security, data protection, legal, risk and audit staff. Consider creating a privacy impact assessment and security plan prior to finalising any designs or going live.
Recent research and regulator opinion suggest that pseudonymous data should be treated as identifiable, rather than truly anonymous. Your privacy approach needs to be properly understood and demonstrable.
When exploring a new idea for growing, combining or manipulating data, you need to be mindful as to whether you need additional consent, as the data you wish to use may have been provided for a different purpose. Having data for one reason, doesn’t automatically allow reuse for another.
Data protection law can be both complex and disparate, especially in an international context, and you may find there are challenging and conflicting requirements. Potential obligations, e.g. the right to be forgotten, also need to be considered in the design and build of any big data system.
Be mindful that as well as rules governing the collection and processing of data, there are others that may require you to disclose information, be it to the data subject, law enforcement or other authorities.
Although big data warrants a cautious approach, security, privacy, and compliance obligations should not temper your ambition to deliver such a project, as they should provide a robust capability to support it not sink it.
Gareth Niblett Chairs the Information Security Specialist Group of BCS, The Chartered Institute for IT; provides security, privacy and compliance advice through Blackarts Limited; and tweets as @garethniblett