Encrypting form data (end-to-end encryption)

Previous Next

While each SurveyCTO component uses strong industry-standard methods like at-rest and in-transit encryption to protect data against unauthorized outside access, end-to-end encryption allows you to protect data even from inside access (from authorized users, server administrators, and even the SurveyCTO platform itself).

With this option, form data is encrypted with a public encryption key as soon as a submission is finalized, and from that point onward it is only readable by people who have the private encryption key. You create and manage your own keys, so not even SurveyCTO engineers or server administrators can see the encrypted data you collect: it passes through SurveyCTO cloud systems in an encrypted, unreadable format.

Click here to watch a video on how to encrypt a form.

This is how the end-to-end encryption works in more detail:

As soon as a surveyor marks a filled-out form as "finalized," the form's contents will be encrypted using your public encryption key – except for those fields explicitly marked as publishable (those fields with a "yes" in their publishable column). Fields explicitly marked as publishable are left unencrypted so that they can be conveniently published to cloud services or directly downloaded from the server. (Please note, however, that file attachments – like photos or audio recordings – are always encrypted: even if they are marked as publishable, they cannot be accessed without the private key.)
From that point forward, the form is essentially 'locked'. Questions and answers that were not marked as publishable will be encrypted – meaning the device used to fill out the form will no longer be able to make any changes to or even view the form. In fact, no devices anywhere will be able to open that form again without the private key.
Whenever form data is transmitted via a 3g or other Internet network, it is encrypted in transit using SSL. This is true for all form data regardless of whether the form itself is configured to be encrypted – so, in a sense, encrypted forms are doubly-encrypted (once with your public key, and then again with SSL).
The SurveyCTO server stores the form data, but it remains encrypted and therefore unreadable by the server (and by anyone who might conceivably compromise that server, legally or otherwise). (Again, fields explicitly marked as publishable will remain readable by the server.)
When the form data is downloaded by an end-user using SurveyCTO Desktop, the Data Explorer, or the Download option on the Export tab, it is again doubly-encrypted, with both your public key and the SSL protocol used for secure data transmission.
Desktop, the Data Explorer, and the server download option all store local backups of the form data in local storage, but it remains encrypted and therefore unreadable.
The end-user who downloads the data can use your private encryption key to decrypt and read that data. For added security, this user may choose to never store that private key on an Internet-connected computer (see the topic on working with cold-room computers). If using a "cold-room" computer is not possible, another approach is to store the contents of the private key in a password manager, rather than keeping the key file saved on a computer. Private keys stored in this way can then be pasted into Desktop when you are ready to export your data.

Creating keys

What makes this model of encryption so secure is that it relies on you generating the encryption keys with which data is encrypted; this assures that you and you alone are able to access your data. So the first step is to generate your own public/private encryption key pair, which you can then use to encrypt all of your sensitive forms.

Keys stored as files

To create a new key pair, navigate to the Design tab, scroll down to the Your forms and datasets section, click the Tools option at the very top of the section, and then click the Create new key option. Once you click to open the key generator into a new browser window, you can disconnect from the Internet if you wish to ensure that the key pair is generated and known only locally, on your computer. In the key generator, you will name the key pair, then "download" the private and then public key files ("download" being in quotes because all of this is happening, for privacy, locally in your web browser and not on our server). By default, the key files are named keyname_PRIVATEDONOTSHARE.pem and keyname_public.pem.

keyname_public.pem is the public key used to encrypt data. As the name suggests, it is public – so you don't need to worry about safeguarding it.

keyname_PRIVATEDONOTSHARE.pem is the private key used to decrypt data. This private key you want to guard very closely. In fact, you may want to generate the key pair on a "cold room" computer that is extraordinarily secure (e.g., disconnected from the Internet) and never have other copies of the private key in less secure locations. (To generate a key pair on a cold-room computer, run SurveyCTO Desktop and select Create encryption key pair... from the Offline form tools menu.)

Keys stored as text

Another approach to safeguarding your private key is to store it in a password manager instead of as a file on your computer. To do this, follow the steps above to launch the key generator from your server console, but instead of downloading the private and public key files, select the options to copy the keys to your clipboard. Then you can simply paste the keys into your password manager, one at a time. If you already have existing key files and want to store them as text instead, open your keyname_PRIVATEDONOTSHARE.pem file using a text editor, and copy the entire contents of that file into a password manager. Your key should start with -----BEGIN RSA PRIVATE KEY----- and end with -----END RSA PRIVATE KEY-----. Once you've saved this private key securely in your password manager, you can delete the keyname_PRIVATEDONOTSHARE.pem file.

Keep your private key safe!

While you do want to be very careful with your private key, make sure that you do not lose it. The private key is your only way to decrypt data. If you lose it, you will also lose the ability to decrypt data. Thus, you will need some very secure way to back up or otherwise safeguard your private key against loss. If you have it only on a cold-room computer, for example, and that computer fails – how will you decrypt your data?

Creating an encrypted form

You cannot encrypt a deployed form

Please note: form encryption must be set up when the form is created on the server. You cannot turn on encryption once the form has been deployed.

Once you have a key pair, you are ready to configure your forms for encryption. To start a new encrypted form, just click + and then Start new form in the SurveyCTO server's Design tab, enable the Advanced options, and mark the checkbox to indicate that you want the form to be encrypted; in the next step, upload or paste your public key when prompted (not the private key, which should be kept private and never shared with anyone). Your new form will then include, on its settings worksheet, a copy of your public key. That will then be used to encrypt all finalized forms.

You will design and deploy these encrypted forms just as you do non-encrypted forms, and surveyors will fill them out and submit them in the usual way. The only real difference will come in when you are ready to publish, download, or analyze your data.

Test before deploying

If you make a mistake that affects one of your keys, you risk never being able to access any data collected with that form. For this reason, you should always test to make sure you've set up your keys properly before starting your actual data collection. The easiest way to test this is to simply fill out your form once (it does not matter which platform you use), then use the Data Explorer to view that submission. If everything is set up correctly, you should be able to view that form's encrypted fields.

Accessing encrypted data

Accessing your encrypted data will require use of your private key. When needed, the server console will prompt you for your private key. When using SurveyCTO Desktop to export your encrypted form's data to local export files, you will need to supply the private key when setting the export location. If you're using a key file, you can click the BROWSE FOR KEY FILE button to point Desktop to where it resides on your local system. If you've saved your key in a password manager, first copy the key to your clipboard, then click the PASTE KEY button. Without that private key, Desktop cannot decrypt and export your data. (It can download the data and store it locally, but it cannot decrypt or export it.)

If you are keeping the private key only on a cold-room computer, follow these steps to download, decrypt, and export your data:

On an Internet-connected computer, run SurveyCTO Desktop with Server as the data source and External drive as the data destination. When you download data, it will download and sync to an external hard drive (either a thumb drive or an external hard drive).
Eject the external drive, bring it to the cold-room computer, and plug it in.
Finally, run SurveyCTO Desktop on the cold-room computer with External drive as the data source and Export as the data destination. You will need to supply the private key during this step, either by browsing for the key file or by pasting it from your clipboard.

Keeping parts of your form unencrypted

If you want to publish some parts of an encrypted form to a server dataset (e.g., to feed directly into other forms) or publish some parts to the cloud (e.g., to monitor key indicators via a Google Sheets dashboard), you will need to explicitly mark some fields as publishable so that they will remain unencrypted (by putting a "yes" in their publishable column). Those fields will still be encrypted in transit, but they will be technically readable by the SurveyCTO server. (Please note, however, that file attachments – like photos or audio recordings – are always encrypted: even if they are marked as publishable, they cannot be accessed without the private key.)

See Keeping your data secure for a broader discussion of SurveyCTO data security, including a list of best practices.