Bulk Export API overview
- Michael Kalnicki
- Sam Louk
The bulk data API allows you to call data from the Engaging Networks servers over a secure connection, without using the query builder. The export can be called from a web form that requests account authentication (using a private token), can be automatically transferred to your server via secure copy (SCP), or by calling the data remotely using an HTTPS GET request. There are a number of parameters that can be set, including:
start date
end date
file type (csv, xml, standardcsv, standardxml – these are explained below)
delimiter (csv/standardcsv only)
The data file exported is an incremental file: only data records that have been created or modified within the specified date range are exported.
The API also allows you to remotely submit an import file to the Engaging Networks servers, based on a file format that has previously been set up in your account dashboard.
Data will be fully available and staged at 3 am ET the following day. A day is 12:00 am – 11:59 pm ET.
What data is available?
The bulk data API can only export transactional data. If you wish to export user or hybrid data, you can use the query builder as usual.
The data is batched into export tables, so you are not accessing live data. This means that you should only attempt to export data up to yesterday, since today’s won’t be fully available.
Check your API Option settings if you are not seeing all the data you expect.
Overview of features
There are two main aspects of the export API to consider:
1) Export file format
2) Software dashboard features to permit customization of the data output
File format
The various export types are explained in the table below.
Type parameter | Export type | Comments |
|---|---|---|
csv | csv file | The file contents will reflect any customization options you have specified through the Data API menu in your user dashboard. |
standardcsv | csv file | The export file will be in a standardized format, common to all clients. Apart from campaign and contact reference names, any customization options specified in the Data API menu will be ignored, and tagged form data fields will be exported in our standard order. |
xml | xml feed | The contents of the feed will reflect any customization options you have specified through the Data API menu in your user dashboard. |
standardxml | xml feed | The contents of the feed will be in a standardized format, common to all clients. Apart from campaign and contact reference names, any customization options specified in the Data API menu will be ignored, and tagged form data fields will be exported in our standard order. |
custom filter name | Custom | If a special export filter has been custom-built for your account, it can be invoked by entering the custom filter name as the ‘type’ field. The format of the resulting file will depend on the specifications of your custom export filter. |
We will look at the ‘csv/standardcsv’ format in some detail. One row of transaction data is exported for each supporter activity that has taken place in the chosen date range.
The export file can broadly be divided into three sections, shown schematically below:
Common transaction data segments – this section contains data columns that are common to all transaction types, such as supporter ID, time of activity, activity reference name/ID.
Activity-specific transaction data segments – for each data row, the contents of this section will depend on the specific type of activity the row relates to. For example:
an ’email to target’ row includes: action status, contact first name, contact last name, contact organisation, contact type.
a ‘broadcast’ row includes: email received, email opened, email clicked, email action completed, hard bounce, soft bounce, and unsubscribe.
The last few columns of this section contain technical/tracking data for those records that relate to a page being accessed (e.g. engagingnetworks.app actions or donation pages).
For details of all columns for all activity types, please see the provided spreadsheet.
Default data segments – this section will either consist of:
(a) ‘csv’ export type – default data columns as set up in Default Supporter Record, or a subset of these columns if an export group has been selected (see ‘customisation via dashboard’, below). In this case the total number of columns will therefore vary from client to client.
Or
(b) ‘standardcsv’ export type are the standard Engaging Networks tagged fields, in a set order:
Email Address
Title
First Name
Middle Name
Last Name
Address 1
Address 2
Address 3
City
Region
Postcode
Country
Phone Number
Customizing your export via the dashboard
Customization options are accessible via the Data & Reports > API Options menu:
File options
This is where you can set general export options.
Export Format allows you to select the default file format, e.g. csv or xml
File Delimiter allows you to set the delimiter for csv exports
File Name allows you to set the file name for FTP/SCP exports
Export Version Name should be left as Version 2, the most up-to-date and comprehensive export version. Version 1 is included for backward compatibility for clients that have been using the API export for some time
Apply Custom Reference Names allows you to enable/disable custom reference “override” names
File format
Engaging Networks’ transactional export file format is pre-defined with header names and columns. The settings here allow you to amend the column names or to exclude them entirely from the export.
You can also enable/disable all supporter data columns by checking or unchecking the User Data option. Additionally, you can enable export of Origin Source and your External Reference columns here.
This is a global change, so please confirm this is correct for your data requirements.
Automate export
This allows you to activate a regular auto-run of the export, with the file transferred to a remote SCP server, running daily, every three days, weekly or monthly (equivalent to 30 days).
The protocol used for secure auto-export is Secure Copy (SCP) over SSH ‚- please ensure that your destination server has SCP transfer enabled before initiating the export in Engaging Networks.
The host name for your destination server should include the scp protocol.
For example, if your destination is ‘servername.com’ and you want to use secure copy, you should type ‘scp:servername.com’.
You can also have the export service place the file in a subfolder on your destination server, for example ‘scp:servername.com//subfolder’ We strongly recommend that you use the secure method if you are exporting supporters’ personal data.
File transfer test will send a text file to the destination server. If that test does not work it is an indication that your credentials are incorrect or that the server is not configured as SCP transfer enabled. There are third party sites that allow you test if your server is accessible from the internet.
You can also set a future date on which to start the regular auto-export sequence.
You should also whitelist these IP addresses:
3.19.5.96
3.140.123.11
3.97.146.169
15.222.229.212Override page name
To apply override names to your exports, first go to API Options > File options and check the “Apply Custom Reference Names” checkbox.
The Override page name section provides a listing of all pages, emails and questions/opt-ins in your account, split between four tabs. You are able to assign a new name for each item. The name or value entered will be the value exported in the ‘Campaign ID’ column in the transactional data export instead of its reference name in the software.
Transaction Types
This allows you to select which types of campaign type you want to include in the transactional export. You can select or unselect.
For a list of campaign types click here.
Apply File Filters
This gives you the ability to filter your records based on supporters’ responses to opt-in questions, eg you can choose to exclude any records pertaining to supporters who have chosen not to opt in.
Setting up a basic export
This step is only required if you are exporting the data via HTML web form, or calling the data using a custom-written client application on your server. The key is not required for automated scp transfer.
Log into your subaccount admin dashboard (as opposed to the subaccount user dashboard) and navigate to Hello YOURNAME > Account settings > Tokens. The token needed for the export API is the ‘private’ key. One private key can be created for each account user. Please note that if any private token has already been created in the account, deleting or re-generating the token will prevent any process using the old version of that token from working. It is important to protect your private token as you would your username and password combination, as it gives access to your supporter data.
Optionally set up any customisation in the export service tab in your user account dashboard. Please see the detailed individual help documents for more information. For a basic, default formatted export, no customisation options need to be set.
Automated SCP export method
In your user account dashboard, navigate to the Data & Reports > API Options > Automate Export. Fill in the required fields as detailed above and submit the page. Your recurring export will be scheduled, based on your selected export frequency and start date, and will appear as a task on your jobs monitor page.
HTML form export method
Navigate to the following URL:
https://ca.engagingnetworks.app/ea-dataservice/export.jsp (Toronto)
https://us.engagingnetworks.app/ea-dataservice/export.jsp (Dallas)
It is not necessary to be logged into your account. Make sure the URL begins with https to ensure secure transfer. Enter your private token and optional parameters as shown on the page, and submit the form. The export will be called, and after a short pause will be downloadable via your web browser. Due to the limitations of the https protocol, exports containing very large amounts of data should not be called via this method. Note that the method can only pull data from the past 31 days.
It is also possible to write a client application, in a programming language of your choice, in order to call the data service by sending an https GET request to our server. The volume limitation due to use of the https protocol and the 31-day limit apply to this method also.
Example Export URLs
https://us.engagingnetworks.app/ea-dataservice/export.service?token=YOUR_TOKEN&startDate=03252018&endDate=04152018&type=csv&configTypes=QCB
Transaction format guide
