In response to the aforementioned data security risks, Sansec has built a big data full life cycle security system based on cryptographic technology after many years of research, trials and experiences.

This security system

• Thoroughly extend the cryptography protection of sensitive data  to the application layer.
• Overcome the security drawbacks: "transmitting with TLS and storing with TDE"
• Effectively solve the retrieval and calculation inconvenience of ciphertext data 
• Build an independent third-party data security authority system

Figure1. Big Data Encryption Scheme

Technical Architecture

This solution combines security platform with cryptographic middleware to provide encryption functions for multiple application components and databases in the big data platform. The architecture is shown in the following figure:

Figure2. Technical Architecture of Big Data Encryption Solution

Security platform: located at the core of the entire cryptographic system, it is responsible for:

• Provide hardware-level security protection for the entire cryptographic framework (HSM)
• Key security management based on KMIP (Key Management Interoperability Protocol)
• Provide identity authentication, access management of ciphertext search engine and cryptographic middleware

Cryptographic middleware: The cryptographic middleware is installed in the form of an application-side software agent. This component is transparently embedded and deployed inside of the application, and it realizes the sensitive data encryption and the key secure management through linkage with the security platform. It has several kinds of cryptographic algorithms, including FPE (Format-preserving encryption) and homomorphic encryption algorithms.

Application layer: Big data platforms mostly use various data processing components based on the Hadoop ecosystem, mainly including three parts:

• Data cleaning and message distribution (ETL and KAFKA, etc.)
• Data storage and processing (HDFS, Hive, Hbase, Spark and Flink, etc.)
• Analysis and presentation (BI).

This solution can support more types of application components, quick customization, and adaptation according to practical requirements.

Product Deployment

The security platform includes HSM, key management system, ciphertext search engine and management terminal. These servers are connected to the big data production cluster through the Ethernet and independently deployment in a secure subnet. At the same time, cryptographic middleware is deployed in various clusters.

Figure3. Big Data Encryption Product Deployment

Main Functions

1.High-performance Data Encryption

Support self-developed cryptographic algorithm engines and optimized algorithm applications to achieve high-performance data encryption.

2.Key Security Management

By using HSM to store the root key securely and the KMIP protocol, we achieve the key centralized management of multiple clients.

3.Independent Access Control for Sensitive Data

Authority access control function under the cryptographic system. Under the permission framework of native Hadoop, realize the permission control for an independent third party's sensitive data.

4.Ciphertext Retrieval and Ciphertext Calculation

• Realize accurate, fuzzy and high frequency ciphertext query functions.
• Overcome the bottleneck that ciphertext can only be accurately searched by accurate condition  and restricted fuzzy condition.
• Achieve full-scene, ciphertext retrieval capabilities that are indistinguishable from plaintext. The cryptographic engine has homomorphic algorithm functions such as Paillier and Elgamal algorithms.