The SpatialKey Data Import API allows developers to programmatically create and update CSV data or Shapefiles in SpatialKey without having to login. With the Command Line Tool, typical users of SpatialKey can leverage this functionality without any programming knowledge. You just need to learn a few of the basics and you will be all set to kick off import and update jobs outside of SpatialKey.

Note that the Data Management API Command Line Tool isn’t currently supported when called through a proxy server.

Why use the Data Import API?

SpatialKey makes it very easy to login and import data directly…  so you might ask yourself, why would I need to use the Data Import API? One common use for the Data Import API is to schedule import tasks.  Let’s say that your organization has an internal process that generates a new CSV file for your dataset every night, you could use the Data Import API to automatically pick up that file and submit an overwrite task in SpatialKey. Instead of coming into work in the morning, logging into SpatialKey, submitting the import job, and then working with the data, you can just wake up, login and start working with the refreshed dataset when you use the Data Import API.

Installing the Command Line Tool

To begin, install the Command Line Tool and sample data. (Download .zip file)

Unzip the contents and put in an accessible location – we will need to connect to that location later. For example, I placed my unzipped folder at c:\program files\SKDM.

Command Line Tool files:

  • skdm.exe (executable file to trigger import or update task)
  • SpatialKeyDataManagerConfig.xml (XML descriptor for executable file)
  • ICSharpCode.SharpZipLib.dll (code that .exe file uses)

Sample Data files:

  • 110thCongressionalDistrictShapefile.zip (sample Shapefile)
  • SalesData.csv (Sample CSV file)
  • SalesData.xml (XML descriptor for CSV file import) – for tips on generating an XML descriptor file for your CSV, check out this article

Setting up the Data Manager Config XML file

The Data Manager Config XML file is split into the following sections:

  • Define Organization
  • Authentication
  • Action

Open the SpatialKeyDataManagerConfig.xml file to get started. If you know XML, these sections will be easy to identify – if you don’t know XML, be patient and search through the file until you find these sections and then plug in the required information. If you need an XML refresher, visit http://www.w3schools.com/xml/default.asp.

Define Organization and Cluster

When you define your organization and cluster in the XML file, you are telling the Data Import API where to send your data. Enter your the part of your SpatialKey URL before the “.spatialkey.com” as your Organization Name into the placeholder in the Config XML file. For example, if my URL was Demo.spatialkey.com, I would set up my Config XML file with an organization name of “Demo”. Enter your full URL as the Cluster Name – be sure to include “https:/” at the beginning, and “/” at the end.

XML File Default: <organizationName>xxx</organizationName>
XML with my SpatialKey Organization Name entered: <organizationName>Demo</organizationName>

XML File Default: <clusterDomainUrl>https://xxxx.spatialkey.com/</clusterDomainUrl>
XML with my SpatialKey Cluster Name: <clusterDomainUrl>https://xxxx.spatialkey.com/</clusterDomainUrl>

Finished section for my organization would look like this:

<organizationName>Demo</organizationName>
<clusterDomainUrl>Demo.spatialkey.com/</clusterDomainUrl>

Note: if you license SpatialKey as an on-premise solution (i.e. you have SpatialKey hosted in your own internal environment), you will have to lookup your Cluster Domain URL. In order to look up your cluster, enter “http://[yoursite].spatialkey.com/clusterlookup.cfm” into your browser. For my organization, I would enter http://Demo.spatialkey.com/clusterlookup.cfm.

Authentication

When you authenticate in the XML file, you are telling the Data Import API who you are and giving it a chance to validate your permissions for the defined organization. You can authenticate in a couple of ways: Authenticate with Keys or Authenticate with username/password.

Authenticate with Keys

To authenticate with keys, you need to generate an API Key and retrieve your user ID from within SpatialKey.

Login to SpatialKey, click on “People” tab, and go into the settings for your user.

On this screen, you can view your user id and generate an API key. Plug these values into the XML placeholders in the Config XML file.

Default apiKey in XML file: <apiKey>xxx</apiKey>
XML file with my apiKey entered: <apiKey>8b08e-fcb-42-93-828a2a2</apiKey>

Default userId in XML file: <userId>xxx</userId>
XML file with my userId entered: <userId>8afd7600ce0a462ad0d8a</userId>

You can comment out the username/password from the XML if you authenticate using keys. See conflicts section below for details on why commenting out is suggested.

Finished section for my organization would look like this:

<apiKey>8b08e-fcb-42-93-828a2a2</apiKey>
<userId>8afd7600ce0a462ad0d8a</userId>
<!--
<userName>user@xxx.com</userName>
<password>xxx</password>
-->

Authenticate with Username/Password

To authenticate with username/password, enter your username and password combination into the Config XML file.

Default userName in XML file: <userName>user@xxx.com</userName>
XML file with my userName: <userName>MyEmail@sk.com</userName>

Default password in XML file: <password>xxx</password>
XML file with my password: <password>MyTempPass99</password>

You can comment out the keys from the XML if you authenticate using username/password. See conflicts section below for details on why commenting out is suggested.

Finished section for my organization would look like this:

<!--
<apiKey>xxx</apiKey>
<userId>xxx</userId>
-->
<userName>MyEmail@sk.com</userName>
<password>MyTempPass99</password>

Action

The “actions” section of the Config XML file defines all the actions that will be done when the Command Line Tool is executed.

The “type” element defines what action will be done. The current options are:

  • overwrite – create or overwrite an existing CSV dataset
  • append – create or Append to an existing CSV dataset
  • poly – create or overwrite a shapefile

Overwrite or Append Action

The overwrite and append actions will allow you to create, append to or overwrite an existing CSV dataset. For both of these actions, the action section in the XML file is defined similarly:

  • action name – the name you will specify when running the executable if you want to run only this action
  • type – either overwrite or append
  • dataPath – the path (relative to the config.xml) to the CSV file to upload
  • xmlPath – the path (relative to the config.xml) to the CSV definition XML
  • runAsBackground – true if you want to run the import in the background and return immediately without waiting for completion – defaults to “true”
  • notifyByEmail – true if you want the authenticated user to be notified on completion – defaults to “true”
  • addAllUsers – true if you want all users in your organization to have access to this dataset once imported into SpatialKey – defaults to “false”

Let’s set up an overwrite action for the sample CSV file provided in the SKDM folder.

<action name="csv example">
<type>overwrite</type>
<dataPath>SalesData.csv</dataPath>
<xmlPath>SalesData.xml</xmlPath>
<runAsBackground>true</runAsBackground>
<notifyByEmail>true</notifyByEmail>
<addAllUsers>false</addAllUsers>
</action>

Poly Action

The poly action will allow you to create or overwrite a shapefile. The poly action in the XML file is slightly different depending on whether you are creating a new shapefile or updating an existing.

Create new shapefile:

  • action name – the name you will specify when running the executable if you want to run only this action
  • type – poly
  • dataPath –  the path (relative to the config.xml) of the shapefile – the shape file must be a zip
  • datasetName – the name of the new shapefile
  • datasetId – this field is ignored when a datasetName is defined, it can be removed or commented out when a datasetName is provided (see “Overwrite existing shapefile” section below for instructions on how to get your datasetId)

Overwrite existing shapefile:

  • action name – the name you will specify when running the executable if you want to run only this action
  • type – poly
  • dataPath –  the path (relative to the config.xml) of the shapefile – the shapefile must be a zip
  • datasetName – comment out or remove this line item completely when doing an overwrite of an existing Shapefile, if you leave it in, a new shapefile will be created
  • datasetId – the id of the existing dataset to be overwritten (To get the datasetId for an existing shape dataset, login to SpatialKey, open the settings for your shapefile from the “Manage Data” tab. Click on the “Advanced Settings” option, then click on the “Data Import API” tab. The dataset ID is at the bottom of this screen.)

Note that in the Config XML file, the poly action is commented out. Be sure to remove comments “<!–” and “–>” if you plan to use this action. Let’s set up this action for both creating a new shapefile and overwriting and existing shapefile with the sample shapefile provided in the SKDM folder.

Create new shapefile example:

<action name="shape example">
<type>poly</type>
<dataPath>110thCongressionalDistrictShapefile.zip</dataPath>
<datasetName>110th Congressional District</datasetName>
<!-- <datasetId>xxx</datasetId> -->
</action>

Overwrite an existing shapefile example:

<action name="shape example">
<type>poly</type>
<dataPath>110thCongressionalDistrictShapefile.zip</dataPath>
<!-- <datasetName>110th Congressional District</datasetName> -->
<datasetId>8ab3d821d013ed04</datasetId>
</action>

Another thing to consider when defining actions…

If you are creating a Config XML file that has many actions and you want to create an exception to the main file’s organizationName, clusterDomainUrl (on-premis customers only), apiKey, userId, username, or password, you can do so by adding a command for a specific action.

Simply add a line item (or multiple) in the action

<action name="csv example">
<organizationName>MyOtherOrgName.spatialkey.com/</organizationName>
<type>overwrite</type>
<dataPath>SalesData.csv</dataPath>
<xmlPath>SalesData.xml</xmlPath>
<runAsBackground>true</runAsBackground>
<notifyByEmail>true</notifyByEmail>
</action>

See conflicts section for some important considerations.

Conflicts

Keep this conflict in mind when setting up your Config XML file.

  • When both Keys (apiKey/userId) and username/password are provided for authentication, they keys will be used. This is why it was suggested that one of the other should be commented out.

When creating exceptions to the main file’s apiKey/userId, or username/password within a specific action, keep the above conflict in mind. As an example, if you define the apiKey/userId in the main body of the Config XML file and you enter the username/password in the action as a exception, the apiKey/userId will still be used because it wins the conflict. In order to completely override to originally defined apiKey/userId in this case, you will have to define the username/password and enter a blank apiKey/userId in the action.

Defined in body for Config XML file:
<apiKey>8b08e-fcb-42-93-828a2a2</apiKey>
<userId>8afd7600ce0a462ad0d8a</userId>
Defined in "action" for Config XML file:
<userName>MyEmail@sk.com</userName> 
<password>MyTempPass99</password>
<apiKey></apiKey>
<userId></userId>

Running the Data Management API Command Line Tool

Now for the easy part. Open command-line prompt and navigate to the SKDM directory where you put the unzipped folder. Run “skdm.exe SpatialKeyDataManagerConfig.xml” – you can optionally specify a list of actions to perform. When no actions are specified, all actions from the Config XML file will be run by default.

Running Data Management API Command Line Tool from a Windows workstation?

Command to run all actions:

skdm.exe SpatialKeyDataManagerConfig.xml

Command to run only the “shape example” action:

skdm.exe SpatialKeyDataManagerConfig.xml "shape example" (add quotes if there is a space in the action name)

Running Data Management API Command Line Tool from a Mac workstation? Install Mono on your workstation. (Installation Instructions)

Command to run all actions:

mono skdm.exe SpatialKeyDataManagerConfig.xml

Command to run only the “shape example” action:

mono skdm.exe SpatialKeyDataManagerConfig.xml "shape example"
(add quotes if there is a space in the action name)