Big Data: security, privacy, and compliance

Big data offers us big potential, for both benefits and risks. Although it has great appeal, we need to strike a healthy balance – to achieve maximum benefit for an acceptable level of risk. Security, privacy, and compliance considerations and solutions should be integral to any big data project.


The value of your data and insight grows as your volume does. Big data aggregates significant volumes of information, which leads to an increased interest from attackers and impact if compromised. Aggregation can be due to the accumulation of data or by associations the data enables.

Infrastructure, systems, applications, databases, processes, transactions and audit logs must all be properly secured; limiting access and rights to only those allowed. You must maintain the confidentiality, integrity, and availability of your critical data, against both external and internal threats.

Security needs to be supported by an ability to audit use and deal with misuse, including incident investigation, digital forensics, disciplinary action, and communications plan. You need to be prepared for the worst.


The associations that big data helps us make can also lead to more personal intrusion than might be understood or accepted. Analysing medical history, browsing and buying patterns, communications metadata, and other data sets, can provide quite a complete view of private lives.

When developing a system that will process large volumes of records, especially if they are sensitive, you should also involve your security, data protection, legal, risk and audit staff. Consider creating a privacy impact assessment and security plan prior to finalising any designs or going live.

Recent research and regulator opinion suggest that pseudonymous data should be treated as identifiable, rather than truly anonymous. Your privacy approach needs to be properly understood and demonstrable.


When exploring a new idea for growing, combining or manipulating data, you need to be mindful as to whether you need additional consent, as the data you wish to use may have been provided for a different purpose. Having data for one reason, doesn’t automatically allow reuse for another.

Data protection law can be both complex and disparate, especially in an international context, and you may find there are challenging and conflicting requirements. Potential obligations, e.g. the right to be forgotten, also need to be considered in the design and build of any big data system.

Be mindful that as well as rules governing the collection and processing of data, there are others that may require you to disclose information, be it to the data subject, law enforcement or other authorities.

Although big data warrants a cautious approach, security, privacy, and compliance obligations should not temper your ambition to deliver such a project, as they should provide a robust capability to support it not sink it.

Gareth Niblett Chairs the Information Security Specialist Group of BCS, The Chartered Institute for IT; provides security, privacy and compliance advice through Blackarts Limited; and tweets as @garethniblett

This post originally appeared in the Media Planet Big Data Report which is available as a download (1MB PDF).

ITNow – Securing the Human

The following introduction was originally published in the Information Security section of the BCS ITNow Magazine, Autumn 2012 issue (Volume 54, Issue 3), which was on the topic of Securing the Human:

The human can be both the weakest link and the strongest tool in building and protecting your organisation, says Gareth Niblett, Chair of BCS ISSG.

People are at least as important as technology when it comes to securing business, but sometimes it appears that investments are made in tin with flashing lights more readily than their fleshy operators and users. It is people that design, build, run and use systems. Businesses should remember to invest in their people as well as the latest technology.

Companies talk about how their employees are their biggest asset, but often it is training, education and awareness, including for information security, that gets cut when times are hard.

Also, when ‘downsizing’, roles that should be kept separate, to minimise fraud and other unwanted activities, can be combined to ‘maximise efficiency and
cost saving’.

Even with recent technological advances, social engineering remains a key threat to organisations and their information.

A number of recent data leaks have been facilitated to some degree by tricking the target or their service providers into divulging what should have been restricted information or enabling unauthorised account access or changes.

With the growth of BYOD and cloud services, organisations need to balance awareness of the risks with the benefits of use; employees need to understand the issues related to them and follow any related policies, ensuring that corporate and client information remains in full compliance with any legal, regulatory or contractual obligations.

The human can be both your weakest link and your strongest tool in building and protecting your organisation. If they are treated as part of the solution and you help secure them then they, in turn, can help keep themselves and your business secure.”

ITNow – Hacktivism and Anonymous

The following introduction was originally published in the Information Security section of the BCS ITNow Magazine, Summer 2012 issue (Volume 54, Issue 2), which was on the topic of Hacktivism and Anonymous:

There is a lot more to online activists, such as Anonymous, than the mainstream media would have us believe, says Gareth Niblett, Chair of BCS ISSG.

Just as the media often uses ‘terrorism’ and ‘Al-Qaida’ as shorthand for a broad church of Islamist militants, so too they use ‘hacktivism’ and ‘Anonymous’ to describe a diverse collection of attackers and motives.

This oversimplification means that the true motivations and intents of online vigilantes are not always adequately acknowledged or addressed.

It is too easy to put them in the ‘bad guy’ bucket and overlook well-intentioned, albeit illegal, activities – such as hunting down online sexual predators, exposing authoritarian regimes, protesting greed and corruption, undermining state censorship and so on.

Anonymous affiliated offshoots, such as AnonOps, LulzSec and AntiSec each have different members, objectives and approaches that lead to different types of targets and attacks – some of these are more questionable than others.

As the authorities close in on the more active members of these ‘leaderless’ groups, plenty of others adopt the anonymous concept whether as members of the franchise or simply following in its wake.

Fear of attack

One recent survey of ‘security experts’ concluded that most were concerned about being attacked by Anonymous.

I’m not sure that I’d be most concerned about them, versus say a more subtle state-sponsored long-term infiltration, as although an attack would be embarrassing it would also likely be relatively obvious and possible to quickly assess the impact.

Although I cannot support the illegal actions of Anonymous, I think we should not be too quick in demonising them either because, as in all things relating to human nature, things are often more nuanced than the headlines might lead one to believe.”

A PDF version of the magazine is available online to BCS members at:

ITNow – Upcoming Threats and Countermeasures

The following article was originally published in the Information Security section of the BCS ITNow Magazine, Spring 2012 issue (Volume 54, Issue 1), which was on the topic of Upcoming Threats and Countermeasures:

Gareth Niblett, Chairman of the BCS Information Security Specialist Group, takes a glimpse into the future* to see what emerging threats we might have to contend with.

Cloud computing offers many benefits over the traditional approach of owning, growing and maintaining a network of computers. It also brings complications and risks related to multi-tenancy hosting, off-shoring, consolidation and jurisdiction.

With such growth in this area, attackers are likely to increase their targeting of both cloud-based services and their underlying platforms, as well as leverage their scalable processing power, e.g. to break encryption.

Extraterritorial application of laws, such as the US PATRIOT Act can undermine, for example, even EU data protection rules and trust if a cloud provider is in scope.

The scale of compromises may be massive when they do occur and the scope for forensic examination of complex platforms is severely restricted by contracts, accessibility, capability and jurisdiction; thereby hampering investigations and prosecutions.

Those putting their, and their customer’s data, on cloud services should ensure that they are able to meet their legal, regulatory and contractual obligations to all parties in contracting, operation, incident response and migration.

Ghosts in the wires

So-called advanced persistent threats (APT), or continued state of ignorance (CSI), will continue. These are large-scale compromises, over a long period of time, possibly state-sponsored or condoned, aimed at extracting commercially or personally sensitive information from the targets.

They will reconnoitre, infiltrate, establish a beachhead and back doors, conduct whatever espionage they have in mind and then exfiltrate data. Social engineering / blagging, spoofing, spearphishing, blackmail/bribery, infiltration, break-ins, dumpster diving, bugging, hacking etc. can all be part of the repertoire.

If even journalists can find out information from phones and computers of celebrities, police and politicians, what do you think your chances are against a suitably motivated, technically capable and well-funded attacker?

Use of targeted attacks and malware will grow, sometimes deniable and sometimes for publicity. Stuxnet and Duqu may have been the tip of an iceberg and based on flexible attack toolkits that can be tailored for specific targets and carry specialist payloads, e.g. for espionage or disruption.

Hacktivism, where a specific message is aimed at the target group and often the intent is as much to embarrass and drive home the message as to compromise systems or information, is also gaining traction.

Not forgetting mass attacks, using vulnerabilities that affects popular software and affects many organisations; this is low hanging fruit for an attacker.

Simply keeping patches up-to-date is clearly insufficient to counter this threat, as recent attacks have used multiple zero-day (0-day) vulnerabilities. If you are running a system that holds customers’ personal and financial data, encryption needs to be both utilised and effective.

The adage of defence in depth should be de rigueur, as competent attackers will come quietly, with patience and persistence.

Your organisation needs the right personnel, policies, technology and procedures to detect, repel, contain and respond, in short order, to minimise downtime, data losses, embarrassment and costs when it does occur.

Online blockades

Excuses for filtering and blocking will carrying on growing, adding to the regime of ineffectual prevention of web access to material related to child sexual abuse, terrorism, religious and racial hatred, sedition, defamation and libel as well as blocking offshore services providing online gambling, copyright infringement and sale of controlled or duty-paid goods, such as drugs, alcohol and tobacco.

Laws and agreements with unintended consequences are the result of addressing the symptoms rather than root causes, i.e. sources – SOPA, PIPA, HADOPI, ACTA, as well as court orders requiring ISPs to block.

Breaking DNS, for example, to render websites inaccessible, undermines the security of the internet and fails in the face of technical competence. Only by removing content or services can they be truly blocked.

International rules should recognise the cross-border nature of the internet and understand that local laws might not work well in an environment that doesn’t have physical borders.

We need better agreements, cooperation and action in tackling harmful and illegal online content and services; those in child safety have had great success in taking down content, services and rescuing those being harmed with international cooperation and assistance from ISPs and others.

Lawmakers should be encouraged not to legislate in areas they don’t understand, especially when based on one-sided arguments put to them by lobbyists. The issue will remain difficult to resolve while there is a disparity in legal and moral positions between countries.

Cyber wargames

The military has traditionally favoured growth and the adoption of new technologies; a bigger empire with more weapons. ‘Cyber’ is no different. The main difference is the unresolved issue of attribution as plausible deniability is much easier in a cyber context.

An attacker’s computers don’t need to mass at a border, nor are they marked so that independent observers can identify them; indeed compromised systems may be used to further obfuscate things.

Cyber arsenals are being built up, not only by the usual suspects of US, UK, Russia and China, but by Israel, Japan and smaller countries.

Cyber warfare capability is a force multiplier and provides the potential for asymmetric warfare. Israel and others have said that they will treat a cyber attacker in the same way as a physical attack and may respond kinetically, rather than simply in kind. Bits vs bombs, if you will.

Rules of cyber conflict, broadly similar to traditional rules, are starting to emerge. This needs to continue and ensure that some of the trickier questions are answered.

Surveillance nation

The false zero sum game of security vs. privacy will continue to play out, with the enforced information sharing of personal and financial data, in the name of protecting life and liberty.

As everyone is a suspect, who needs to be monitored to ensure compliance, we need a panopticon for all communications – so say an increasing number of governments, trying to suck up everything to find needles.

Without transparency, oversight and being made to justify and temper ambitions, governments, law enforcement and intelligence agencies will continue to take liberties with our privacy, and abuses will stay hidden, unless citizens take a stand.

Trap and trade

Where miscreants are identified, we will see increased use of mutual legal assistance, international arrest warrants and extradition even where the suspect may not have committed an offence in the jurisdiction where they are sought from.

Nationals that would not be extradited by their home country will be tricked into, or tracked, travelling to more co-operative countries where they are then seized.

This will not just be for criminal hackers and fraudsters but also whistleblowers, copyright infringers and online gambling site operators etc.

International rules in this area should be equitable and extradition requests should be based on the same level of evidence required for a prosecution in the country holding the subject in question; preferably for a crime recognised there also.

Media Muddle

Writers will continue to conflate and confuse denial of service with hacking and downloading with sharing.

The fear mongers will repeat long in the tooth scare stories about terrorist hackers taking down all the technology and services we take for granted.

There will often be seeds of truth and fact behind the dark sinister tales, but should not be taken as gospel without supporting evidence.

Identity matters

UK citizens will fail to realise the benefits of a robust service-focussed digital identity, due to past government mistakes on the national ID card, whereas, in the US, NSTIC may create a two-tier internet in relation to online identity.

ID cards and online identities need not be bad things, provided the intentions of those providing them can be trusted and the necessary checks and balances are in place and used.

Problems arise when your information is gathered without clear informed consent, analysed, sold and used ‘against’ you. Users can gorge on cookies, becoming the product that is marketed to advertisers, whilst adding increasing amounts of personal detail and value to their social profiles.

Use of social logins and sharing will exacerbate this. ‘Do not track’ will fail to gain significant traction, due to technical reasons and user adoption, as have other web privacy and security standards.

What can be done? Well, I don’t think people are going to suddenly stop using Google, Facebook, Twitter and LinkedIn, but the new EU Data Protection reforms may well finally make non-EU entities play fair with data belonging to EU citizens.

You’re unique

People share passwords between accounts. It is human nature, especially when long and complex passwords are required.

The brain can only cope with so much, and there’s only so much space for Post-Its. If you register with any online service you need to be mindful that many attacks have exposed client information (including personal details, passwords and credit cards).

This can lead to further compromises, especially when an email address is used as the user ID and the password is reused.

Use a password manager. Long, Unique, Complex and Kept safe – you make your own LUCK. Password reminders and resets are often sent to your registered email account.

Never use your email password with another online service; otherwise you may be gifting an attacker with another route in. If you have control of your own domain name(s) and email accounts, you have the option of using a different email address for each online service.

Combined with unique logins, this would mean that a single compromise doesn’t undermine all your accounts.

*Risk can increase as well as decrease. Past security performance does not guarantee future success. We suggest you seek advice from a qualified security advisor etc.

A PDF version of the magazine is available online to BCS members at:

ISNow – Digital Loss Prevention

The following introduction was originally published in the BCS Information Security Now Magazine, Winter 2011 issue (Volume 6, Issue 1), which was on the topic of Digital Loss Prevention:

Gareth Niblett, Chair of the ISSG, says that many people see DLP software as something that magically stops your data from being lost, whereas the reality is quite different. 

DLP is often used as a catch-all term for technology that somehow magically stops all unauthorised information flows once it has been installed.

In reality DLP has much wider implications in an organisation and can be quite nuanced. It could be considered as part of information lifecycle management, and should focus on ensuring the organisation can share the information it needs to, both internally and externally, in a correct, accountable and secure manner – data loss is then also prevented as a beneficial by-product.

To achieve robust inter-organisational collaboration capabilities, we would need common policies for identity proofing and verification (IPV) of organisations, people and devices, issuance of credentials, authentication, authorisation so that interoperability can be obtained (for consistency and cost reasons).

Add a bridge, to tie together disparate systems and organisations with cross certification, along with an independent verification process, to ensure assurance is provided to all parties. Mix.

Levels of trust

One leading initiative, which I am involved with, is working towards such federated trust, at higher levels of assurance between regulated companies and industries. The British Business Federation Authority (BBFA) is a not-for-profit self-regulating organisation born out of a government request for a body to represent the needs of UK industry in relation to identity management, which came at a joint BCS/eema seminar in 2009.

The BBFA Steering Group is made up of companies from regulated industry sectors and, along with its policy management authorities, is working with both private and public sector organisations towards standards- based and interoperable IPV, strong authentication and authorisation, federation and PKI bridge policies, procedures and mechanisms, as it recognises that without these no technology can meet the real needs of customers and end users.”

A PDF version of the magazine is available online at:

The Blog of Gareth Niblett