Technical documentation - Importer Service
- Mattias Löfstrand
- Corentin Humbert
- Kenneth Jonsson
Import Service is an automatic function to import Xml file data into Signifikant databases. It is run as a Windows Service and is always active on a server. The same server may be available in the Signifikant Server and in the Signifikant Web Viewer, but with separate configuration.
The service acts on new files, Files and handles them in a pipe line - files are transformed into internal data format - data is uploaded to a remote server or stored locally.
Xml file transformations are configurable through scripts, in case no further programming is needed to convert a new custom Xml file format into the internal format accepted for upload and stored into a database. The remote server Api is called using REST calls, at HTTP port 80, to reduce the need to setup a network to access remote server.
The service do not import documents, images, illustrations. video or other media data. It is intended to update Signifikant databases, either the editor database or the server database.
This service is available in version 5.1 and later
Overview
The import process require a configuration of folders, filenames, transformers, handlers, logging etc,. Once this is setup the service is working automatically to handle new data.
Data flow and process:
A new file is generated on a predefined disk area by an external system
The service supervises the disk area and detects the new file, the file name is compared to a name pattern to determine if the file is handled and how it is handled
The import handler is called to transform the file to an internal format suitable for upload
The items loaded to the importer is passed to the uploader
The uploader handler calls a remote server Api, passing a chunk of items
The remote Api will handle the data and store it to the server, the REST call indicates what interface is called and type of the data to store
Installation
Run the installer.
Possibly register the service, see Windows Service installer finishes correctly but service is missing.
Ensure Signifikant Service is run under “Local System” or an account which has access to SQL Server.
Configuration
Imported data
Currently we handle part prices, part availability, organisations and customers - all is imported to the Server database on a remote server. It is also possible to import part information into a Manager site.
The service configuration is in: C:\ProgramData\Signifikant\Assert\importer-service.config. Each configuration is defined in a CustomFolderSettings tag and, if it is to upload to a web server, refers an UploaderId depending on upload action.
UploaderId | Usage |
|---|---|
PartsPrice | Import prices for parts per price list into Server database. |
PartsBalance | Import availability for parts per warehouse into Server database. |
Parts | Import ReplacementCode and AvailabilityCode on parts. |
Organizations | Import organizations and addresses for organizations into Server database. |
Users | Import users into Server database. |
AssetUpload | Import assets into Asset database. |
Invoice | Import invoices into Server database. |
Sample configuration
<?xml version="1.0" encoding="utf-8"?>
<CustomFolderConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Enabled>true</Enabled>
<!-- In case SaveItem is true, importer service will save imports directly to the database. In this case importer service will
need to know where to find the server.config file since it may be either in AppData or in Program data depending on if importer
service is installed on Manager Server or on Web Server. -->
<ServerConfigPath>C:\ProgramData\Signifikant\Assert\server.config</ServerConfigPath>
<!-- Log files placed here. -->
<LogFolder>C:\Signifikant\Log\ImportLog</LogFolder>
<!-- File watcher will be recreated every x hrs. Default is 0 which means no recreation. Setting available from version 5.1.7 and later. -->
<SuperviseHours>48</SuperviseHours>
<Settings>
<CustomFolderSettings ImporterId="Parts">
<DumpUploadItems>true</DumpUploadItems>
<FolderDescription>Part status</FolderDescription>
<FolderEnabled>true</FolderEnabled>
<FolderFilter>Parts*.xml</FolderFilter>
<FolderPath>C:\Signifikant\Import_service</FolderPath>
<BackupFolder>C:\Signifikant\Import_service_backup</BackupFolder>
<FolderIncludeSub>false</FolderIncludeSub>
<MoveToBackup>true</MoveToBackup>
<SaveItems>true</SaveItems>
<XlsSheet/>
<SsxSheet>signifikant_parts.xsl</SsxSheet>
<UploadItems>true</UploadItems>
<UploadServer>http://localhost/AssertWeb/</UploadServer>
<UploaderId>Parts</UploaderId>
<UploadSiteId>Alo</UploadSiteId>
<UploadLogin>false</UploadLogin>
<UploadUsername>uploader</UploadUsername>
<UploadPassword>uploader</UploadPassword>
<Options>
<Option key="PurgeAfter" value="true"/>
</Options>
</CustomFolderSettings>
<CustomFolderSettings ImporterId="PartsPrice">
<DumpUploadItems>true</DumpUploadItems>
<FolderDescription>Part price</FolderDescription>
<FolderEnabled>true</FolderEnabled>
<FolderFilter>PartsPrice*.xml</FolderFilter>
<FolderPath>C:\Signifikant\Import_service</FolderPath>
<BackupFolder>C:\Signifikant\Import_service_backup</BackupFolder>
<FolderIncludeSub>false</FolderIncludeSub>
<MoveToBackup>true</MoveToBackup>
<SaveItems>true</SaveItems>
<XlsSheet/>
<SsxSheet>signifikant_price.xsl</SsxSheet>
<UploadItems>true</UploadItems>
<UploadServer>http://localhost/AssertWeb/</UploadServer>
<UploaderId>PartsPrice</UploaderId>
<UploadSiteId>Alo</UploadSiteId>
<UploadLogin>false</UploadLogin>
<UploadUsername>uploader</UploadUsername>
<UploadPassword>uploader</UploadPassword>
<Options>
<Option key="PurgeAfter" value="true"/>
</Options>
</CustomFolderSettings>
<CustomFolderSettings ImporterId="PartsBalance">
<FolderEnabled>true</FolderEnabled>
<FolderDescription>Part availability</FolderDescription>
<FolderFilter>PartsBalance*.xml</FolderFilter>
<FolderPath>C:\Signifikant\Import_service</FolderPath>
<BackupFolder>C:\Signifikant\Import_service_backup</BackupFolder>
<FolderIncludeSub>false</FolderIncludeSub>
<MoveToBackup>true</MoveToBackup>
<SaveItems>true</SaveItems>
<SsxSheet>signifikant_avaibility.xsl</SsxSheet>
<UploadItems>true</UploadItems>
<UploadServer>http://localhost/AssertWeb/</UploadServer>
<UploaderId>PartsBalance</UploaderId>
<UploadSiteId>Alo</UploadSiteId>
<UploadLogin>false</UploadLogin>
<UploadUsername>uploader</UploadUsername>
<UploadPassword>uploader</UploadPassword>
<Options>
<Option key="PurgeAfter" value="true"/>
</Options>
</CustomFolderSettings>
<CustomFolderSettings ImporterId="Organizations">
<FolderEnabled>true</FolderEnabled>
<FolderDescription>Organization import</FolderDescription>
<FolderFilter>Organisations*.xml</FolderFilter>
<FolderPath>C:\Signifikant\Import_service</FolderPath>
<BackupFolder>C:\Signifikant\Import_service_backup</BackupFolder>
<FolderIncludeSub>false</FolderIncludeSub>
<MoveToBackup>true</MoveToBackup>
<SaveItems>true</SaveItems>
<SsxSheet>signifikant_organization.xsl</SsxSheet>
<UploadItems>true</UploadItems>
<UploadServer>http://localhost/AssertWeb/</UploadServer>
<UploaderId>Organizations</UploaderId>
<UploadSiteId>Alo</UploadSiteId>
<UploadLogin>false</UploadLogin>
<UploadUsername>uploader</UploadUsername>
<UploadPassword>uploader</UploadPassword>
<Options>
<Option key="PurgeAfter" value="true"/>
</Options>
</CustomFolderSettings>
<CustomFolderSettings ImporterId="Users">
<FolderEnabled>true</FolderEnabled>
<FolderDescription>User Import</FolderDescription>
<FolderFilter>Users*.xml</FolderFilter>
<FolderPath>C:\Signifikant\Import_service</FolderPath>
<BackupFolder>C:\Signifikant\Import_service_backup</BackupFolder>
<FolderIncludeSub>false</FolderIncludeSub>
<MoveToBackup>true</MoveToBackup>
<SaveItems>true</SaveItems>
<SsxSheet>signifikant_user.xsl</SsxSheet>
<UploadItems>true</UploadItems>
<UploadServer>http://localhost/AssertWeb/</UploadServer>
<UploaderId>Users</UploaderId>
<UploadSiteId>Alo</UploadSiteId>
<UploadLogin>false</UploadLogin>
<UploadUsername>uploader</UploadUsername>
<UploadPassword>uploader</UploadPassword>
<Options>
<Option key="PurgeAfter" value="false"/>
</Options>
</CustomFolderSettings>
<CustomFolderSettings ImporterId="Invoice">
<FolderEnabled>true</FolderEnabled>
<FolderDescription>Initial data place holder and template (Invoices for Dometic)</FolderDescription>
<FolderFilter>Invoice*.xml</FolderFilter>
<FolderPath>C:\Signifikant\Import_service</FolderPath>
<DeleteAfterImport>true</DeleteAfterImport>
<FolderIncludeSub>false</FolderIncludeSub>
<SaveItems>true</SaveItems>
<XlsSheet>signifikant_invoice.xsl</XlsSheet>
<SsxSheet />
<UploadItems>true</UploadItems>
<UploaderId>Invoice</UploaderId>
<UploadServer>http://localhost/AssertWeb/</UploadServer>
<UploadUsername>uploader</UploadUsername>
<UploadPassword>uploader</UploadPassword>
<UploadLogin>false</UploadLogin>
<Options>
<Option key="PurgeAfter" value="false"/>
</Options>
</CustomFolderSettings>
</Settings>
</CustomFolderConfiguration>Desrciption of tags
Tags | Description | Options |
|---|---|---|
ServerConfigPath | Path to the server.config file where configuration is kept for the editor- or web-server. Important is the Notification settings, and the integration to Monitoring dashboard. | For an editor server: C:\ProgramData\Signifikant\Assert\server.config. For a web server: C:\inetpub\wwwroot\AssertWeb\App_Data |
FolderEnabled | If configuration is active or not. | true/false |
FolderDescription | Description. | Text |
FolderFilter | Regex used on the files that appear in the folder. Only files matching the regex will be handled, e.g. Part_price*.xml. | Text |
FolderPath | Folder path to monitor for new files. |
|
FolderIncludeSub | If sub folders are to be scanned for imports. | Text |
MaxIdleDays | Time limit when it is considered an error if no new import files arrive, see below. | Integer |
MaxIdleHours | Time limit when it is considered an error if no new import files arrive, see below. | Integer |
DeleteAfterImport | If file should be deleted after a successful import. (NB: This takes precedence over backup settings) Note. This option is available in version 5.2 and later. | true/false |
MoveToBackup | If file is to be moved to backup folder. | true/false |
BackupFolder | After an import or upload is completed, move file to this folder. |
|
SaveItems | If file content is to be saved on database on same server as importer service runs on. | true/false |
SiteId | Site name when importing into Manager site. | Site name |
UploadItem | If file content is to be uploaded to web server site. | true/false |
UploadSiteId | Site name when uploading into web viewer site. | Site name |
UploadServer | Url for the web server e.g https://www.domain.com/AssertWeb/ |
|
UploaderId | Type of upload, see table above. | See table above |
UploadLogin | If login required to upload items to webserver. | true/false |
UploadUsername | User name and password for calling the upload API on the web server. Api permissions are setup in: C:\ProgramData\Signifikant\Assert\api-permission.config, see Technical documentation - Signifikant REST API authentication . |
|
UploadPassword |
| |
DumpUploadItems | If upload dump is to be generated in log folder. | true/false |
XlsSheet | Custom Xls for the importer. Custom Xls shall be placed in C:\ProgramData\Signifikant\Assert\Customize |
|
SsxSheet | Signifikant predefined SSX format for the import type as signifikant_organization.xsl |
|
Options | Used to define alternatives to control the import | See separate table below. |
Options tag has a set of keys.
Key | Value | Description | Options |
|---|---|---|---|
PurgeAfter | bool | After last call with all uploaded part numbers, a valid number list with all valid part numbers are uploaded. All parts in the price list or warehouse lists which not in the valid numbers list removed from that price list or warehouse list. That way removed prices/warehouse stock will get deleted. Use this method if full price lists or warehouse lists are sent. |
|
Customization
Custom style sheet may be used to transform input format. Use tag <SsxSheet> to define a custom xsl.
Handlers for import or upload to remote server may be added in a customize module. By referring the custom handler in service configuration, any customer use case regarding automated file import, can be implemented. Custom modules require programming.
Permissions
The remote server may require permissions to access the upload Api interface. In that case we use basic authentication with a username/password to secure the function.
In this case the service need to declare login parameters for an upload handler, in the service configuration.
Logging
Configuration declares folders for logging, where each handler appends text to one file per day. Example of a log file name: UploadService_20200528_PartsPrice.log, indicating price information was uploaded.
The service also logs to an application log file, normally found at: C:\ProgramData\Signifikant\Assert\ImportService.log
Mail and event notification
Setting is available in 5.1.1 and later. Notification is made by mail, by monitoring event or both.
Notification by mail
Time limit when it is considered an error if no new import files arrive. If no import files arrives within the defined number of hours and days, a notification is created. The notification is defined in server.config.
<Notification>
<Type>Service</Type>
<MailEnabled>true</MailEnabled>
<MailRecipient>support@company.com</MailRecipient>
<MailSender>noreply@company.com</MailSender>
<MailAtSuccess>false</MailAtSuccess>
<MailAtFailure>true</MailAtFailure>
</Notification>If none of MaxIdleHours and MaxIdleDays are defined, no monitoring will be turned on.
Ensure to use the server.config defined in tag ServerConfigPath in importer-service.config.
Notification by event generation to monitoring dashboard
To send import events to the monitoring dashboard, it uses a configuration file monitoring-service-client.json. The file is located in the same folder as the file defined by the ServerConfigPath element. Ensure that the file is in the same folder as the active server.config file. Should the file be missing. an empty file is generated, as a template for configuration. It has the enabled attribute defined to false to disable event generation initially.
Example of file monitoring-service-client.json, apiKey is received from Signifikant, externalId and customer may be used to identify events later.
{
"enabled": true,
"url": "https://development.signifikant.se/monitoringserviceapi/",
"apiKey": "*",
"defaults": {
"externalId": "id",
"customer": "name"
},
"apiKeyHeaderName": "api-key",
"sendUrl": "api/event"
}
Custom Xsl for uploads
Overview
Xsl files transform customer exported files in one Xml format to a format that the import/upload engine handles. File names are entered in the configuration file and the file is located in common or custom program data folders.
Prices
Input format
Standard input format:
Output format
The price conversion Xsl file generates output records, one for each parts price for a price list.
If a part has a price list which does not exist, that price list will be created and made available. It will not be a default price list.
Availability
Input format
Standard input format:
Output format
The availability conversion Xsl file generates output records, one for each parts balance for a warehouse. The total of balance + supply-balance is loaded to the database - 137 in the example below.
If a part has a warehouse which does not exist, that warehouse will be created and made available. It will not be a default warehouse.
Part status
Input format
The part status import is used to set the AvailabilityCode and ReplacementCode on parts. Standard input format:
Organizations
Input format
For the file format, refer to definition of the Signifikant import format for organization. Standard input format.
Users
Input format
Here is an example of users xml.
Assets
Currently there is no transformation for the asset data. To import assets it is likely that a customize module is required to maintain all business data in exports. One simple example of an asset looks like.
Settings for testing
Version 5.1 and later
Importer service has a few extra settings intended for testing purposes, which may be used for error search.