Project Item Development

This document discusses the basics of project item development: what is required make one, how items interact with the Toolbox GUI and how they are executed.

The core of every project item consists of two classes: a static project item class which is responsible for integrating the item with the Toolbox GUI and an executable class which does the item’s ‘thing’ and exists only during execution in Spine Engine. Some additional classes are needed for Toolbox to be able to instantiate project items and to communicate with the user via the Toolbox GUI.

Specifications are a way to make the settings of an item portable across projects. In a sense a specification is a template that can specialize an item for a specific purpose such as a Tool that runs certain model with known inputs an outputs. Items that support specifications need to implement some additional methods and classes.

Getting started

Probably the most convenient way to start developing a new project item is to work with a copy of some simple project item. For example, View provides a good starting point.

Project items are mostly self-contained Python packages. It is customary to structure the project item packages like the Toolbox itself: mvcmodels submodule for Qt’s models, ui module for automatically generated UI forms and widgets for widgets’ business logic. However, the only actual requirement is that Toolbox expects to find the item’s factory and item info classes in the package’s root modules as well as an executable_item module.

Item info

A subclass of spine_engine.project_item.project_item_info.ProjectItemInfo must be found in one of the root modules of an item’s package. It is used by Toolbox to query the type and category of an item. Type identifies the project item while category is used by the Toolbox GUI to group project items with similar function. Categories are currently fixed and can be checked from spine_items.category.

Item Factory

The details of constructing a project item and related objects have been abstracted away from Toolbox by a factory that must be provided by every project item in a root module of the item’s package. The factory is a subclass of spinetoolbox.project_item.project_item_factory.ProjectItemFactory. Note that methods in the factory that deal with specifications need to be implemented only by items that support them.

Executable item

A project item must have a root module called executable_item that contains a class named ExecutableItem which is a subclass of spine_engine.project_item.executable_item_base.ExecutableItemBase ExecutableItem acts as an access point to Spine Engine and contains the item’s execution logic.

Toolbox side project item

A project item must subclass spinetoolbox.project_item.project_item.ProjectItem and return the subclass in its factory’s item_class() method. Also make_item() must return an instance of this class. This class forms the core of integrating the item with Toolbox.

Specifications

Items that support specifications need to subclass spine_engine.project_item.project_item_specification_factory.ProjectItemSpecificationFactory which provides an access point to Toolbox and Spine Engine to generate specifications. The factory must be called SpecificationFactory and be placed in specification_factory module under item package’s root. The specification itself should be a subclass of spine_engine.project_item.project_item_specification.ProjectItemSpecification.

Toolbox GUI integration

ProjectItemFactory.icon() returns a URL to the item’s icon resource. This is the item’s ‘symbol’ shown e.g. on the main toolbar of Toolbox. It should not be confused with the actual icon on Design view which in turn is a subclass of spinetoolbox.project_item.project_item_icon.ProjectItemIcon and is returned by ProjectItemFactory.make_icon().

When creating a new item on the Design view Toolbox shows the Add item dialog it gets from ProjectItemFactory.make_add_item_widget(). Toolbox provides spinetoolbox.widgets.add_project_item_widget.AddProjectItemWidget which is a general purpose widget for this purpose though project items are free to implement their own widgets as needed.

Once the item is on the Design view, the main interaction with it goes through the properties widget which is created by ProjectItemFactory.make_properties_widget(). The properties widget should have all controls needed to set up the item.

Saving and restoring project items

Project items are saved in JSON format as part of the project.json file. Item saving is handled by ProjectItem.item_dict() which should return a JSON compatible dict and contain at least the information returned by the base class method.

File system paths are handled specifically during saving: all paths outside the project directory should be absolute while the paths in the project directory should be relative. This is to enable self-contained projects which include all needed files and can be easily transferred from system to system. As such, paths are saved as special dictionaries. spine_engine.utils.serialization.serialize_path(), spine_engine.utils.serialization.serialize_url() and spine_engine.utils.serialization.deserialize_path() help with dealing with the paths.

ProjectItem.from_dict() is responsible for restoring a saved project item from the dictionary. ProjectItem.parse_item_dict() can help to deserialize the basic data needed by the base class.

Passing data between items: resources

Project items share data by files or via databases. One item writes a file which is then read by another item. Project item resources are used to communicate the URLs of these files and databases.

Resources are instances of the spine.engine.project_item.project_item_resource.ProjectItemResource class.

Both static items and their executable counterparts pass resources. The major difference is that static item’s may pass resource promises such as files that are generated during the execution. The full path to the promised files or even their final names may not be known until the items are executed.

During execution resources are propagated only to item’s direct predecessors and successors. Static items offer their resources to direct successors only. Resources that are communicated to successor items are basically output files that the successor items can use for input. Currently, the only resource that is propagated to predecessor items is database URLs by Data Store project items. As Data Stores leave the responsibility of writing to the database to other items it has to tell these items where to write their output data.

The table below lists the resources each project item type provides during execution.

Item	Notes	Provides to predecessor	Provides to successor
Data Connection	1	n/a	File URLs
Data Store	2	Database URL	Database URL
Data Transformer	3	n/a	Database URL
Exporter		n/a	File URLs
Importer		n/a	n/a
Merger		n/a	n/a
Tool	4	n/a	File URLs
View		n/a	n/a

1

Data connection provides paths to local files.

2

Data Store provides a database URL to direct successors and predecessors. Note, that this is the only project item that provides resources to it’s predecessors.

3

Data Transformer provides its predecessors’ database URLs modified by transformation configuration embedded in the URL.

4

Tool’s output files are specified by a Tool specification.

The table below lists the resources that might be used by each item type during execution.

Item	Notes	Accepts from predecessor	Accepts from successor
Data Connection		n/a	n/a
Data Store		n/a	n/a
Data Transformer		Database URL	n/a
Exporter		Database URL	n/a
Importer	5	File URLs	Database URL
Merger		Database URL	Database URL
Tool	6	File URLs, database URLs	Database URLs
View		Database URLs	n/a

5

Importer requires a database URL from its successor for writing the mapped data. This can be provided by a Data Store.

6

Tool specification specifies tool’s optional and required input files. Database URLs can be passed to the tool program via command line arguments but are otherwise ignored by the Tool project item. Currently, there is no mechanism to know if a URL is actually required by a tool program. For more information, see Tool Specification Editor.

Execution

Spine Engine instantiates the executable items in a DAG before the execution starts. Then, Engine declares forward and backward resources for each item using ExecutableItemBase.output_resources(). During execution, ExecutableItemBase.execute() is invoked with lists of available resources if an item is selected for execution. Otherwise, ExecutableItemBase.exclude_execution() is called.