Introduction to advanced dataset usage
SurveyCTO datasets help you to organize and manage your data. Broadly speaking, you can use datasets to:
Provide pre-loaded data as an input into one or more survey forms.
Organize, combine, monitor, export, and publish subsets of the data submitted by one or more survey forms.
You can combine the first two functions for situations where you want to treat the data submitted as part of one or more survey forms as pre-loaded data for one or more other survey forms. As you can imagine, there are a great many possibilities.
Datasets are similar to spreadsheets. They are organized around rows (also known as "records"), on the one hand, and columns (also known as "fields") on the other. SurveyCTO datasets (also known as "server datasets") are constructed and maintained on the server. They can be used to pre-load data into survey forms, and form submission data can publish into these datasets (however: in the case of encrypted forms, only fields explicitly flagged as publishable – and thus left un-encrypted – can be published into a server dataset). Server datasets can also be monitored for data quality or published to the cloud so that incoming data streams out to, e.g., some kind of outside visualization or dashboard.
For a small working example, see the "dataset basics" sample form.
Types of server dataset
SurveyCTO currently supports three types of server dataset:
All datasets are similar in that they are collections of rows and columns, can be attached to forms, and can be published into by forms. Enumerator and cases datasets are only different in that they each have a specific structure in terms of their columns, have some special settings, and integrate into the broader platform in particular ways. So, for example, the enumerator field type loads enumerator lists from enumerator datasets, and the Manage Cases interface loads case lists from cases datasets. See Managing enumerators and Case management for details.
Manual use of server datasets
The Your forms and datasets section of the Design tab has all of the options you need to manually manage server datasets for the purposes of attaching pre-loaded data to your survey forms. See Pre-loading data into a form for a full discussion. You can upload new or revised data; download, rename, or purge existing data; and manage to which forms each dataset will be attached as pre-loaded data.
When you purge a server dataset's existing data, all rows of data will be deleted – but the existing columns in that dataset will remain (albeit currently empty). If you want to completely eliminate old columns that are no longer desired, you will need to delete the dataset entirely, then re-create it.
And when you upload data for a dataset, you always upload a .csv or an .xlsx file. Please note the following:
The first row of your .csv or .xlsx file should include short, unique names for each column. These column names should not themselves include commas or quotes. Any uploaded column names that do not correspond to fields already in the dataset will be added to the dataset as new fields.
If your data contains non-English fonts or special characters, you will need to save your .csv file in Unicode/UTF-8 format. Please note, if you upload an .xlsx file instead of .csv, the character encoding will be converted automatically. If you cannot directly save or export either an .xlsx file or a .csv file in Unicode/UTF-8 format, you can use SurveyCTO Desktop to re-encode it: choose Re-encode .csv from the Offline form tools menu, select your file and the encoding for which its text appears correctly in the preview window, and then click Convert to save the re-encoded .csv file.
When you upload new data for an existing dataset, you can choose whether to append the new data to the dataset's existing data, merge with existing data, or replace all existing data. If you choose to merge with existing data, you specify the name of one of your .csv or .xlsx columns to use in uniquely identifying dataset records; incoming .csv or .xlsx rows will either update an existing row of the dataset – if a row with a matching value in that column already exists – or insert a new row into the dataset. When specifying a column with which to merge, there must be both (a) a column in the incoming .csv or .xlsx file with the specified name, and (b) a field in the existing dataset by that same name.
Aside from attaching server datasets to survey forms, you can also download or export these datasets for your own back-office or analytical purposes. To download a dataset's current data, click on Download and then Download data on your server console's Design tab; that will give you a .csv file with all of the dataset's current data. To export a dataset's current data, simply select that dataset from the list of forms and datasets when exporting data with SurveyCTO Desktop. Exporting dataset data follows the same process as exporting form data. Please note that if you are automatically publishing form submissions into your dataset, then your download or export may not reflect data published within the last few minutes. Dataset .csv files are only updated once 5-10 minutes have passed since the last update to the dataset; this prevents datasets – and the forms to which they are attached – from updating with every single form submission.
And finally, you can click Attach to manage the list of forms to which any of your server datasets are attached. An attached dataset's data will be available as pre-loaded data that can be pulled into calculated form fields, used to dynamically populate multiple-choice option lists, or even pre-loaded as default values for user-editable survey fields.
That covers the basics of using manually-managed datasets for attaching pre-loaded data to survey forms. See the following topics in this section for discussions of more advanced dataset techniques.