Tag Archives: privacy

Big Data: security, privacy, and compliance

Big data offers us big potential, for both benefits and risks. Although it has great appeal, we need to strike a healthy balance – to achieve maximum benefit for an acceptable level of risk. Security, privacy, and compliance considerations and solutions should be integral to any big data project.


The value of your data and insight grows as your volume does. Big data aggregates significant volumes of information, which leads to an increased interest from attackers and impact if compromised. Aggregation can be due to the accumulation of data or by associations the data enables.

Infrastructure, systems, applications, databases, processes, transactions and audit logs must all be properly secured; limiting access and rights to only those allowed. You must maintain the confidentiality, integrity, and availability of your critical data, against both external and internal threats.

Security needs to be supported by an ability to audit use and deal with misuse, including incident investigation, digital forensics, disciplinary action, and communications plan. You need to be prepared for the worst.


The associations that big data helps us make can also lead to more personal intrusion than might be understood or accepted. Analysing medical history, browsing and buying patterns, communications metadata, and other data sets, can provide quite a complete view of private lives.

When developing a system that will process large volumes of records, especially if they are sensitive, you should also involve your security, data protection, legal, risk and audit staff. Consider creating a privacy impact assessment and security plan prior to finalising any designs or going live.

Recent research and regulator opinion suggest that pseudonymous data should be treated as identifiable, rather than truly anonymous. Your privacy approach needs to be properly understood and demonstrable.


When exploring a new idea for growing, combining or manipulating data, you need to be mindful as to whether you need additional consent, as the data you wish to use may have been provided for a different purpose. Having data for one reason, doesn’t automatically allow reuse for another.

Data protection law can be both complex and disparate, especially in an international context, and you may find there are challenging and conflicting requirements. Potential obligations, e.g. the right to be forgotten, also need to be considered in the design and build of any big data system.

Be mindful that as well as rules governing the collection and processing of data, there are others that may require you to disclose information, be it to the data subject, law enforcement or other authorities.

Although big data warrants a cautious approach, security, privacy, and compliance obligations should not temper your ambition to deliver such a project, as they should provide a robust capability to support it not sink it.

Gareth Niblett Chairs the Information Security Specialist Group of BCS, The Chartered Institute for IT; provides security, privacy and compliance advice through Blackarts Limited; and tweets as @garethniblett

This post originally appeared in the Media Planet Big Data Report which is available as a download (1MB PDF).

Talk on ‘RIPA: Perception & Practice’ (Slides)

I spoke at the BCS Information Security Specialist Group’s 11th Annual Legal Day, held at the RAF Club, London on Friday 22nd January 2010. My talk was entitled “RIPA: Perception and Practice” and the slides are now available:
View more documents from INFOSEC_Maven.

Talk on ‘RIPA: Perception & Practice’

I will be speaking at the BCS Information Security Specialist Group’s 11th Annual Legal Day, being held at the RAF Club, London on Friday 22nd January 2010. My talk is entitled “RIPA: Perception and Practice”, although I was tempted to title it “RIPA: Proportionality, Paranoia and Practice”, and the synopsis is:

“There has been much discussion in the media and elsewhere about the use and misuse of the powers granted to many public authorities under the Regulation of Investigatory Powers Act 2000 and associated legislation.

Stories about snooping on people for trying to get their children into a particular school or letting their dogs foul the street may make the front page, but they are not necessarily representative of how the powers are used in general.

Sure, they should lead to questions about the implementation and effectiveness of the necessity and proportionality tests that are a mandatory part of the legislation, but there may be greater things to concern ourselves with when law enforcement and the intelligence community wish to grow and extend the use of data retention, monitoring and surveillance.

This talk will give an overview of many years of practical experience and interactions with the public authorities authorised to seek access to information under RIPA, Part I, Chapters I & II.”

Further information can be found in the events section of the BCS ISSG website:


ISNow – Information Privacy

The following introduction was originally published in the BCS Information Security Now Magazine, Summer 2007 issue (Volume 1, Issue 4), which was on the topic of Information Privacy:

Watching big brother
The government is your friend. If you are doing nothing wrong, you have nothing to fear. We only want to help keep you safe. Big brother knows best. If only it were that simple.

We have more CCTV cameras per capita than any other country along with automatic number plate recognition for congestion charging and alerting police to infractions.

We are building a massive centralised national ID database that will store more than is necessary for us to prove who we are, and offer little in way of consumer benefits.

Our national DNA database has records on over four million people and growing. Police can indefinitely retain the DNA data of anyone they arrest – even if they are never charged or convicted of an offence.

Not only do we fingerprint suspects and criminals, but schoolchildren, without proper guidelines, parental communications or informed consent. Before long, you will need to be fingerprinted to obtain a UK passport.

Soon there will be a mandatory regime of data retention for telephone calls, text messages, mobile location, internet access, emails and web logs. Retained data can be requested by many organisations, for a variety of reasons.

Combined with interception of communications, RFID passports, facial recognition, suspicious behaviour heuristics, satellite car tracking, personnel vetting and dubious data sharing practices you may wonder where the real benefits are and whether we are in a surveillance society.

Work life
Life at work can be ruled by policies on the use of company resources. Your activity and communications may be recorded, archived and monitored for disciplinary or compliance reasons. You leave any expectations of privacy at the door when you arrive.

Unfortunately, work cannot always be left behind when you finish for the day. Employers may respond (negatively) to something you say in your online diary or are seen doing in a picture on a photo sharing website. Recruiters may also conduct a search on you.

Personal life
It may be personal, but life is becoming less private, with profiling (of searches and purchases), sharing (of financial, insurance and health information), tracking (of journeys, transactions and communications), monitoring (of the rubbish in your bins) and enforcement (of TV licences and road tax) and much more besides.

Things can only get better?
I hope so. Hopefully, the opposing forces of security and privacy will come to a happy equilibrium, without one negating the other. More efforts need to be made on privacy enhancing technologies, breach notification and setting standards (for security, retention period, format, quality and so on) where data is kept and disclosure.”

A PDF version of the magazine is available online at: