The SpatialKey Data Management REST API allows developers to programmatically create and update CSV data or Shapefiles in SpatialKey without having to login. With the Command Line Tool, typical users of SpatialKey can leverage this functionality without any programming knowledge. You just need to learn a few of the basics and you will be all set to kicking off import and update jobs outside of SpatialKey.
Why use the Data Management API?
SpatialKey makes it very easy to login and manage data directly… so you might ask yourself, why would I need to use the Data Management API? One common use for the Data Management API is to schedule import tasks. Let’s say that your organization has an internal process that generates a new CSV file for your dataset every night, you could use the Data Management API to automatically pick up that file and submit an overwrite task in SpatialKey. Instead of coming into work in the morning, logging into SpatialKey, submitting the import job, and then working with the data, you can just wake up, login and start working with the refreshed dataset when you use the Data Management API.
Steps to using the Command Line Tool
There are a few important steps that this document will walk you through. Here is a quick outline of them before diving into the details below.
- Installing the Command Line Tool
- Setting up the Data Manager Config XML file
- Gathering files for import
- Running the Command Line Tool
To begin, install the Command Line Tool and sample data. (Download .zip file) Unzip the contents and put in an accessible location – we will need to connect to that location later. For example, I placed my unzipped folder at c:\program files\SKDM.
Command Line Tool files:
Sample Shapefile files:
Sample CSV files:
Sample Insurance Files:
– Sample Locations.csv
– Sample Policies.csv
– Sample Locations and Policies Insurance with peril specific limits.xml
– Sample Locations and Policies Insurance.xml
The Data Manager Config XML file is split into the following sections:
See the sample SpatialKeyDataManagerConfig.xml shipped with the application for the proper xml format. If you know XML, these sections will be easy to identify – if you don’t know XML, be patient and search through the file until you find these sections and then plug in the required information. If you need an XML refresher, visit W3Schools XML Tutorial.
In order to authenticate to the api, you must first setup your organization URL. This should be the same URL you use to access the client up to and including “.spatialkey.com”. Then you will need to get the organization API and secret key.
To finish authenticating, you need to get your user API key.
Define Proxy (optional)
You likely don’t need to worry about this section because we’ll auto-detect a Proxy for you, but in the event that we cannot auto-detect a proxy, you may need to work with your IT department to provide additional information to the Command Line Tool.
You may need to define a proxy if your IT department has implemented one for your company and not set it up on your machine. If a proxy is in use, the IT department likely set up your OS/default browser to use it. Setting proxyEnabled to true (or just letting it default) will cause skdm to use the default proxy. If you try to use the Command Line Tool and we cannot auto-detect the proxy if needed or you don’t have it manually defined, you’ll get a error message.
These fields define what proxy server to use.
- <proxyEnable> defaults to true. If “false”, no proxy will be used (not even the default proxy).
- <proxyURL> and <proxyPort> define a custom proxy to use. If either are blank, skdm will use the default proxy.
- <proxyUser> and <proxyPassword> are used for custom proxy that requires authentication if both are not blank.
- <proxyDomain> is used for the custom proxy and is needed for NTLM and Kerberos
<proxyEnabled>true</proxyEnabled> <proxyURL>127.0.0.1</proxyURL> <proxyPort>8888</proxyPort> <proxyUser>proxyusername</proxyUser> <proxyPassword>proxypassword</proxyPassword> <proxyDomain>proxydomain</proxyDomain> Note that leaving this code out of the XML will result in <proxyEnabled>true</proxyEnabled> and the rest will be auto-detected if needed.
The “actions” section of the Config XML file defines all the actions that will be done when the Command Line Tool is executed.
|Note that datasetId is required for the Append or Overwrite actions. If the Id isn’t provided, the action will be treated as a regular Import. If a datasetId is provided but the Import actionType selected, the datasetId will be ignored.
Note: new datasets that are imported will count towards your dataset quota, whereas append and overwrite just adjust the existing file.
|dataType||CSV, Shapefile, Insurance||CSV and Shapefile refer to the actual data types. Insurance is a specific type of CSV import where insurance fields are specifically identified in the accompanying XML file.|
|pathData||Name of the dataset to load including extension, e.g. SalesData.csv.|
|pathXML||Name of the accompanying XML file including extension, e.g. SalesData.xml.|
|datasetId||The Id of the existing dataset to be overwritten or appended to. This can be found in SpatialKey in the settings for a given dataset (see image below) or by using the “List” command in the Command Line Tool, [skdm.exe list]. If the Dataset Id isn’t provided, the action will be treated as a regular Import.|
Note: you don’t get email or client notifications at this time.
Initial Import CSV Dataset (could have also used <actionType>append</actionType>)
<action name="csv example"> <actionType>overwrite</actionType> <dataType>CSV</dataType> <pathData>SalesData.csv</pathData> <pathXML>SalesData.xml</pathXML> <datasetId></datasetId> </action>
Overwrite Existing CSV Dataset
<action name="csv example"> <actionType>overwrite</actionType> <dataType>CSV</dataType> <pathData>SalesData.csv</pathData> <pathXML>SalesData.xml</pathXML> <datasetId>1111-848-33-33-283838</datasetId> </action>
Append To Existing CSV Dataset
<action name="csv example"> <actionType>append</actionType> <dataType>CSV</dataType> <pathData>SalesData.csv</pathData> <pathXML>SalesData.xml</pathXML> <datasetId>1111-848-33-33-283838</datasetId> </action>
Overwrite existing Shapefile
<action name="shape example"> <actionType>overwrite</actionType> <dataType>Shapefile</dataType> <pathData>
110thCongressionalDistrictShapefile.xml</pathXML> <datasetId>1111-848-33-33-283838</datasetId> </action>
Exceptions to Organization and User definitions
If you are creating a Config XML file that has many actions and you want to create an exception to the main file’s organizationURL, organizationAPIKey, organizationSecretKey, or userAPIKey, you can do so by adding a command for a specific action. Simply add a line item (or multiple) in the action.
<action name="csv example"> <organizationURL>https://xxx.spatialkey.com/</organizationURL> <organizationAPIKey>xxx</organizationAPIKey> <organizationSecretKey>xxx</organizationSecretKey> <userAPIKey>xxx</userAPIKey> <actionType>overwrite</actionType> <dataType>CSV</dataType> <pathData>SalesData.csv</pathData> <pathXML>SalesData.xml</pathXML> <datasetId>1111-848-33-33-283838</datasetId> </action>
When creating exceptions to the main file’s organizationName, organizationURL, organizationAPIKey, or userAPIKey within a specific action, remember that any not defined in the action will used the default ones.
DatasetId for Overwrite or Append
If you are doing an overwrite or append but haven’t defined a datasetId, the application will do an import instead. If you defined /no-wait, the datasetId will not be updated.
InsuranceId for Overwrite or Append
If you are working with an insurance dataset and want to overwrite or append to it, you need to first obtain the insurance id which is created when the insurance dataset is first uploaded into SpatialKey. Please contact us if you want to perform this action and we can help you get set up.
Append not allowed for Shapefile upload
Note that you can only upload new Shapefiles or Overwrite existing Shapefiles. You cannot append to Shapefiles.
You’ll want to be sure that you have appropriate permissions for various actions you take via the Command Line Tool.
- Must have appropriate permissions to import datasets to the selected organization. It’s possible that a user is blocked for performing imports. The best way to test if you run into issues is whether a user can import directly into SpatialKey without using the Command Line Tool.
- Must have appropriate permissions for a dataset in order to update or append to it. You can check if you have permission to update or append to a dataset by going to the dataset’s setting in SpatialKey and seeing if you have the option to update or append to it there.
- Must have appropriate permissions for a dataset in order to delete it. You can check if you have permission to delete to a dataset by going to the dataset’s setting in SpatialKey and seeing if you have the option to delete to it there.
- If the organization you are importing to has reached the dataset limit cap, the Command Line Tool will throw an error.
- If the dataset that you are importing exceeds the organization limits (shapefile size or number of records in the CSV), the Command Line Tool will throw an error.
What type of data can be imported into SpatialKey?
- CSV datasets
- Insurance datasets (these consist of two CSV files, one for the locations and another for the corresponding policies)
When importing data through the DMAPI, a XML descriptor file is used in conjunction with the data you want to import. For each of the types of data listed above, different information is required in the XML file. You can either use the shortcut method of XML file generation or jump in both both feet and create an XML file from scratch. The shortcut method is quite simple, start by importing your dataset into SpatialKey – after your data is imported into SpatialKey, a complete XML file is just a few clicks away. Find your dataset in the Datasets list, click the gear icon to view data settings, and select “Data Import API”.
You will have the option to “generate API config” on the following screen.
After the XML is generated, save the XML file to the same location as your CSV file and then zip the files together. Note: when saving the XML file, save it directly from the browser page that it appears in – do not copy and paste into another file.
Now for the easy part. Open command-line prompt and navigate to the SKDM directory where you put the unzipped folder. Run “skdm.exe SpatialKeyDataManagerConfig.xml” – you can optionally specify a list of actions to perform. When no actions are specified, all actions from the Config XML file will be run by default. For additional documentation and examples, click here.
Running Data Management API Command Line Tool from a Windows workstation?
skdm.exe [global option(s)] <command> [command option(s)] [argument]
Running Data Management API Command Line Tool from a Mac workstation?
Install Mono on your workstation. (Installation Instructions)
mono skdm.exe [global option(s)] <command> [command option(s)] [argument]
What are global options, commands, command options, and arguments?
Note that all examples are in Windows syntax.
Options available in the Command Line Tool are listed below. The only option that most users will need is the config option – the other options are most useful in debugging situations.
|/help||Returns a list of options.|
|/version||Returns version of Command Line Tool.|
|/trace LEVEL||Trace adds verbosity to the Command Line Tool. Valid LEVEL options: null, 0, 1, 2.
|/config NEWCONFIGFILENAME.XML||Specifies which config xml file to use for the command. The default config xml is “SpatialKeyDataManagerConfig.xml”, anything other than this name will need to be specified view the config option. Note that the config option can be used in conjunction with Commands and Arguments.|
Commands and Arguments
Commands available in the Command Line Tool are listed below. The upload, list and delete commands are the most common – the other commands are slightly more advanced as they provide alternative ways to accomplish dataset xml generation and authentication, which already have the preferred workflow defined in this document (see above sections for XML generation and Authentication).
Arguments should be defined when only select items are intended to be specified. E.g. Run the specified upload action in SpatialKeyDataManagerConfig.xml.
skdm.exe upload "sample csv" (add quotes when the action name has spaces in it)
Note that Arguments are optional and can be used with the upload and suggest commands only.
|Commands||Command Options (defined in section below)||Arguments||Description|
|[ACTION(S)]||Triggers processing of all listed [ACTION(S)] from the config xml file. Listing no [ACTION(S)] will trigger processing of all actions in the config xml.|
|suggest||[ACTION(s)]||Returns xml for dataset(s) referenced in listed [ACTION(S)] from the config xml file. If the xml suggested by <pathXML> doesn’t exist, the xml will be written to that file name. If the file does exist, the xml will be written to a new file. Listing no [ACTION(S)] returns xmls for all datasets referenced in the listed actions in the config file. Note this isn’t suggested for use with insurance dataType at this time.|
|list||Returns list of dataset ids for the organization identified in the config xml.|
|delete||[DATASETID(s)]||Runs a delete action on the identified dataset id(s) for the organization and user defined in the config xml.|
|oauth||/ttl [TTL]||Get oAuth key for organization and user defined in config xml.|
- /no-wait – doesn’t wait for import, append or overwrite action to complete.
- /keep-uploadid – don’t cancel the upload id so it can be used in other manual operations
- /ttl – sets the time to live for the token. The default is 60 seconds.
Command Line Examples
Run all upload actions in SpatialKeyDataManagerConfig.xml
Run the specified upload action in SpatialKeyDataManagerConfig.xml
skdm.exe upload /no-wait "sample csv"
Run all upload actions in AnoterConfig.xml
skdm.exe /config AnotherConfig.xml upload
Run suggest the default pathXML config for all actions in SpatialKeyDataManagerConfig.xml
List all datasets for the organziation and user defined in SpatialKeyDataManagerConfig.xml
Delete the given dataset at the organization and user defined in SpatialKeyDataManagerConfig.xml
skdm.exe delete 55555-66-88-33-abcd
Get oAuth key for organization and user defined in in SpatialKeyDataManagerConfig.xml
skdm.exe oauth /ttl 600