spinetoolbox.spine_io.exporters.gdx

For exporting a database to GAMS .gdx file.

Currently, this module supports databases that are “GAMS-like”, that is, they follow the EAV model but the object classes, objects, relationship classes etc. directly reflect the GAMS data structures. Conversions e.g. from Spine model to TIMES are not supported at the moment.

This module contains low level functions for reading a database into an intermediate format and for writing that intermediate format into a .gdx file. A higher lever function to_gdx_file() that does basically everything needed for exporting is provided for convenience.

author:

1. Soininen (VTT)

date:

30.8.2019

Module Contents

exception spinetoolbox.spine_io.exporters.gdx.GdxExportException(message)

Bases: Exception

An exception raised when something goes wrong within the gdx module.

Parameters:message (str) – a message detailing the cause of the exception

message

A message detailing the cause of the exception.

__str__(self)

Returns the message detailing the cause of the exception.

class spinetoolbox.spine_io.exporters.gdx.Set(name, description='', domain_names=None)

Represents a GAMS domain, set or a subset.

description

set’s explanatory text

Type:str

domain_names

a list of superset (domain) names, None if the Set is a domain

Type:list

name

set’s name

Type:str

records

set’s elements as a list of Record objects

Type:list

Parameters:

• name (str) – set’s name
• description (str) – set’s explanatory text
• domain_names (list) – a list of indexing domain names

dimensions

Number of dimensions of this Set.

is_domain(self)

Returns True if this set is a domain set.

to_dict(self)

Stores Set to a dictionary.

static from_dict(set_dict)

Restores Set from a dictionary.

static from_object_class(object_class)

Constructs a Set from database’s object class row.

Parameters:object_class (namedtuple) – an object class row from the database

static from_relationship_class(relationship_class)

Constructs a Set from database’s relationship class row.

Parameters:relationship_class (namedtuple) – a relationship class row from the database

class spinetoolbox.spine_io.exporters.gdx.Record(keys)

Represents a GAMS set element in a Set.

Parameters:

• keys (tuple) – a tuple of record’s keys
• keys – a tuple of record’s keys

name

Record’s ‘name’ as a comma separated list of its keys.

__eq__(self, other)

Returns True if other is equal to self.

Parameters:other (Record) – a record to compare to

to_dict(self)

Stores Record to a dictionary.

static from_dict(record_dict)

Restores Record from a dictionary.

static from_object(object_)

Constructs a record from database’s object row.

Parameters:object (namedtuple) – an object or relationship row from the database

static from_relationship(relationship)

Constructs a record from database’s relationship row.

Parameters:relationship (namedtuple) – a relationship row from the database

class spinetoolbox.spine_io.exporters.gdx.Parameter(domain_names, indexes, values)

Represents a GAMS parameter.

domain_names

indexing domain names (currently Parameters can be indexed by domains only)

Type:list

indexes

parameter’s indexes

Type:list

values

parameter’s values

Type:list

Parameters:

• domain_names (list) – indexing domain names (currently Parameters can be indexed by domains only)
• indexes (list) – parameter’s indexes
• values (list) – parameter’s values

__eq__(self, other)

append_value(self, index, value)

Appends a new value.

Parameters:

• index (tuple) – record keys indexing the value
• value – a value

append_object_parameter(self, object_parameter)

Appends a value from object parameter.

Parameters:object_parameter (namedtuple) – an object parameter row from the database

append_relationship_parameter(self, relationship_parameter)

Appends a value from relationship parameter.

Parameters:relationship_parameter (namedtuple) – a relationship parameter row from the database

slurp(self, parameter)

Appends the indexes and values from another parameter.

Parameters:parameter (Parameter) – a parameter to append from

is_scalar(self)

Returns True if this parameter contains only scalars.

is_indexed(self)

Returns True if this parameter contains only indexed values.

expand_indexes(self, indexing_setting)

Expands indexed values to scalars in place by adding a new dimension (index).

The indexes and values attributes are resized to accommodate all scalars in the indexed values. A new indexing domain is inserted to domain_names and the corresponding keys into indexes. Effectively, this increases parameter’s dimensions by one.

Parameters:indexing_setting (IndexingSetting) – description of how the expansion should be done

static from_object_parameter(object_parameter)

Constructs a GAMS parameter from database’s object parameter row

Parameters:object_parameter (namedtuple) – a parameter row from the database

static from_relationship_parameter(relationship_parameter)

Constructs a GAMS parameter from database’s relationship parameter row

Parameters:relationship_parameter (namedtuple) – a parameter row from the database

static from_entity_class_parameter_definition(entity_class)

Constructs an empty GAMS parameter from database’s parameter definition row

Parameters:entity_class – a parameter definition row from the database

class spinetoolbox.spine_io.exporters.gdx.IndexingDomain(name, description, indexes, pick_list)

This class holds the indexes that should be used for indexed parameter value expansion.

name

indexing domain’s name

Type:str

description

domain’s description

Type:str

Picks the keys from base_domain for which the corresponding element in pick_list holds True.

Parameters:

• name (str) – indexing domain’s name
• description (str) – domain’s description
• indexes (list) – a list of indexing key tuples
• pick_list (list) – a list of booleans

indexes

a list of picked indexing key tuples

all_indexes

a list of all indexing key tuples

pick_list

list of boolean values where True means the corresponding index should be picked

sort_indexes(self, settings)

Sorts the indexes according to settings.

Parameters:settings (Settings) – a Settings object

to_dict(self)

Stores IndexingDomain to a dictionary.

static from_dict(domain_dict)

Restores IndexingDomain from a dictionary.

static from_base_domain(base_domain, pick_list)

Builds a new IndexingDomain from an existing Set.

Parameters:

• base_domain (Set) – a domain set that holds the indexes
• pick_list (list) – a list of booleans

spinetoolbox.spine_io.exporters.gdx.sort_indexing_domain_indexes(indexing_settings, settings)

Sorts the index keys of an indexing domain in place.

Parameters:

• indexing_settings (dict) – a mapping from parameter name to IndexingSetting
• settings (Settings) – settings

spinetoolbox.spine_io.exporters.gdx._python_interpreter_bitness()

Returns 64 for 64bit Python interpreter or 32 for 32bit interpreter.

spinetoolbox.spine_io.exporters.gdx._read_value(value_in_database)

Converts a parameter from its database representation to a value object.

spinetoolbox.spine_io.exporters.gdx._windows_dlls_exist(gams_path)

Returns True if required DLL files exist in given GAMS installation path.

spinetoolbox.spine_io.exporters.gdx.find_gams_directory()

Returns GAMS installation directory or None if not found.

On Windows systems, this function looks for gams.location in registry; on other systems the PATH environment variable is checked.

Returns:a path to GAMS installation directory or None if not found.

spinetoolbox.spine_io.exporters.gdx.expand_indexed_parameter_values(parameters, indexing_settings)

Expands the dimensions of indexed parameter values.

Parameters:

• parameters (dict) – a map from parameter names to Parameters.
• indexing_settings (dict) – mapping from parameter name to IndexingSetting

class spinetoolbox.spine_io.exporters.gdx.MergingSetting(parameter_names, new_domain_name, new_domain_description, previous_set, previous_domain_names)

Holds settings needed to merge a single parameter.

parameter_names

parameters to merge

Type:list

new_domain_name

name of the additional domain that contains the parameter names

Type:str

new_domain_description

explanatory text for the additional domain

Type:str

previous_set

name of the set containing the parameters before merging; not needed for the actual merging but included here to make the parameters’ origing traceable

Type:str

Parameters:

• parameter_names (list) – parameters to merge
• new_domain_name (str) – name of the additional domain that contains the parameter names
• new_domain_description (str) – explanatory text for the additional domain
• previous_set (str) – name of the set containing the parameters before merging
• previous_domain_names (list) – list of parameters’ original indexing domains

domain_names(self)

Composes a list of merged parameter’s indexing domains.

Returns:a list of indexing domains including the new domain containing the merged parameters’ names Return type:list

to_dict(self)

Stores the settings to a dictionary.

static from_dict(setting_dict)

Restores settings from a dictionary.

spinetoolbox.spine_io.exporters.gdx.update_merging_settings(merging_settings, settings, db_map)

Returns parameter merging settings updated according to new export settings.

Parameters:

• merging_settings (dict) – old settings to be updated
• settings (Settings) – new gdx export settings
• db_map (spinedb_api.DatabaseMapping) – a database map

Returns:

merged old and new merging settings

Return type:

dict

spinetoolbox.spine_io.exporters.gdx.merging_domain(merging_setting)

Constructs the additional indexing domain which contains the merged parameters’ names.

spinetoolbox.spine_io.exporters.gdx.merge_parameters(parameters, merging_settings)

Merges multiple parameters into a single parameter.

Note, that the merged parameters will be removed from the parameters dictionary.

Parameters:

• parameters (dict) – a mapping from existing parameter name to its Parameter object
• merging_settings (dict) – a mapping from the merged parameter name to its merging settings

Returns:

a mapping from merged parameter name to its Parameter object

Return type:

dict

spinetoolbox.spine_io.exporters.gdx.sets_to_gams(gdx_file, sets, omitted_set=None)

Writes Set objects to .gdx file as GAMS sets.

Records and Parameters contained within the Sets are written as well.

Parameters:

• gdx_file (GdxFile) – a target file
• sets (list) – a list of Set objects
• omitted_set (Set) – prevents writing this set even if it is included in given sets

spinetoolbox.spine_io.exporters.gdx.parameters_to_gams(gdx_file, parameters)

Writes parameters to .gdx file as GAMS parameters.

Parameters:

• gdx_file (GdxFile) – a target file
• parameters (dict) – a list of Parameter objects

spinetoolbox.spine_io.exporters.gdx.domain_parameters_to_gams_scalars(gdx_file, parameters, domain_name)

Adds the parameter from given domain as a scalar to .gdx file.

The added parameters are erased from parameters.

Parameters:

• gdx_file (GdxFile) – a target file
• parameters (dict) – a map from parameter name to Parameter object
• domain_name (str) – name of domain whose parameters to add

Returns:

a list of non-scalar parameters

spinetoolbox.spine_io.exporters.gdx.object_classes_to_domains(db_map)

Converts object classes, objects and object parameters from a database to the intermediate format.

Object classes get converted to Set objects while objects are stored as Records in corresponding DomainSets. Lastly, object parameters are read into Parameter objects.

Parameters:db_map (spinedb_api.DatabaseMapping) – a database map Returns:a tuple containing list of Set objects and a dict of Parameter objects

spinetoolbox.spine_io.exporters.gdx.relationship_classes_to_sets(db_map)

Converts relationship classes, relationships and relationship parameters from a database to the intermediate format.

Relationship classes get converted to Set objects while relationships are stored as SetRecords in corresponding Sets. Lastly, relationship parameters are read into Parameter objects.

Parameters:db_map (spinedb_api.DatabaseMapping) – a database map Returns:a tuple containing a list of Set objects and a dict of Parameter objects

spinetoolbox.spine_io.exporters.gdx.domain_names_and_records(db_map)

Returns a list of domain names and a map from a name to list of record keys.

Parameters:db_map (spinedb_api.DatabaseMapping) – a database map Returns:a tuple containing list of domain names and a dict from domain name to its records

spinetoolbox.spine_io.exporters.gdx.set_names_and_records(db_map)

Returns a list of set names and a map from a name to list of record keys.

Parameters:db_map (spinedb_api.DatabaseMapping) – a database map Returns:a tuple containing list of set names and a dict from set name to its records

class spinetoolbox.spine_io.exporters.gdx.IndexingSetting(indexed_parameter)

Settings for indexed value expansion for a single Parameter.

parameter

a parameter containing indexed values

Type:Parameter

indexing_domain

indexing info

Type:IndexingDomain

index_position

where to insert the new index when expanding a parameter

Type:int

Parameters:indexed_parameter (Parameter) – a parameter containing indexed values

append_parameter(self, parameter)

Adds indexes and values from another parameter.

spinetoolbox.spine_io.exporters.gdx.make_indexing_settings(db_map)

Constructs skeleton indexing settings for parameter indexed value expansion.

Parameters:db_map (spinedb_api.DatabaseMapping) – a database mapping Returns:a mapping from parameter name to IndexingSetting Return type:dict

spinetoolbox.spine_io.exporters.gdx.update_indexing_settings(old_indexing_settings, new_indexing_settings, settings)

Returns new indexing settings merged from old and new ones.

Entries that do not exist in old settings will be removed. If entries exist in both settings the old one will be chosen if both entries are ‘equal’, otherwise the new entry will override the old one. Entries existing in new settings only will be added.

Parameters:

• old_indexing_settings (dict) – settings to be updated
• new_indexing_settings (dict) – settings used for updating
• settings (Settings) – new gdx export settings

Returns:

merged old and new indexing settings

Return type:

dict

spinetoolbox.spine_io.exporters.gdx.indexing_settings_to_dict(settings)

Stores indexing settings to a JSON compatible dictionary.

Parameters:settings (dict) – a mapping from parameter name to IndexingSetting. Returns:a JSON serializable dictionary

spinetoolbox.spine_io.exporters.gdx.indexing_settings_from_dict(settings_dict, db_map)

Restores indexing settings from a json compatible dictionary.

Parameters:

• settings (dict) – a JSON compatible dictionary representing parameter indexing settings.
• db_map (DatabaseMapping) – database mapping

Returns:

a dictionary mapping parameter name to IndexingSetting.

spinetoolbox.spine_io.exporters.gdx._find_parameter(parameter_name, db_map)

Searches for parameter_name in db_map and returns Parameter.

spinetoolbox.spine_io.exporters.gdx.filter_and_sort_sets(sets, sorted_set_names, metadatas)

Returns a list of sets sorted by sorted_set_names and their filter flag set to True

This function removes the sets that are not supposed to be exported and sorts the rest according to the order specified by sorted_set_names.

Parameters:

• sets (list) – a list of sets (DomainSet or Set) to be filtered and sorted
• sorted_set_names (list) – a list of set names in the order they should be in the output list, including ones to be removed
• metadatas (list) – list of SetMetadata objects in the same order as sorted_set_names;

Returns:

a list of sets

Return type:

list

spinetoolbox.spine_io.exporters.gdx.sort_records_inplace(sets, settings)

Sorts the record lists of given domains according to the order given in settings.

Parameters:

• sets (list) – a list of DomainSet or Set objects whose records are to be sorted
• settings (Settings) – settings that define the sorting order

spinetoolbox.spine_io.exporters.gdx.extract_domain(domains, name_to_extract)

Extracts the domain with given name from a list of domains.

Parameters:

• domains (list) – a list of Set objects
• name_to_extract (str) – name of the domain to be extracted

Returns:

a tuple (list, Set) of the modified domains list and the extracted Set object

spinetoolbox.spine_io.exporters.gdx.to_gdx_file(database_map, file_name, additional_domains, settings, indexing_settings, merging_settings, gams_system_directory=None)

Exports given database map into .gdx file.

Parameters:

• database_map (spinedb_api.DatabaseMapping) – a database to export
• file_name (str) – output file name
• additional_domains (list) – a list of extra domains not in the database
• settings (Settings) – export settings
• indexing_settings (dict) – a dictionary containing settings for indexed parameter expansion
• merging_settings (dict) – a list of merging settings for parameter merging
• gams_system_directory (str) – path to GAMS system directory or None to let GAMS choose one for you

spinetoolbox.spine_io.exporters.gdx.make_settings(database_map)

Builds a Settings object from given database.

Parameters:database_map (spinedb_api.DatabaseMapping) – a database from which domains, sets, records etc are extracted Returns:a Settings object useful for exporting the given database_map

class spinetoolbox.spine_io.exporters.gdx.Settings(domain_names, set_names, records, domain_metadatas=None, set_metadatas=None, global_parameters_domain_name='')

This class holds some settings needed by to_gdx_file() for .gdx export.

Settings is mostly concerned about the order in which domains, sets and records are exported into the .gdx file. This order is paramount for some models, like TIMES.

Constructs a new Settings object.

Parameters:

• domain_names (list) – a list of Set names
• set_names (list) – a list of Set names
• records (dict) – a mapping from Set names to record key tuples
• domain_metadatas (list) – a list of SetMetadata objects, one for each domain
• set_metadatas (list) – a list of SetMetadata objects, one for each set
• global_parameters_domain_name (str) – name of the Set whose parameters to export as GAMS scalars

sorted_domain_names

this list defines the order in which domains are exported into the .gdx file.

domain_metadatas

this list contains SetMetadata objects for each name in domain_names

sorted_set_names

this list defines the order in which sets are exported into the .gdx file.

set_metadatas

this list contains SetMetadata objects for each name in set_names

global_parameters_domain_name

the name of the domain, parameters of which should be exported as GAMS scalars

add_or_replace_domain(self, domain, metadata)

Adds a new domain or replaces an existing domain’s records and metadata.

Parameters:

• domain (Set) – a domain to add/replace
• metadata (SetMetadata) – domain’s metadata

Returns:

True if a new domain was added, False if an existing domain was replaced

domain_index(self, domain)

Returns an integral index to the domain’s name in sorted domain names.

del_domain_at(self, index)

Erases domain name at given integral index.

update_domain(self, domain)

Updates domain’s records.

sorted_record_key_lists(self, name)

Returns a list of record keys for given domain or set name.

The list defines the order in which the records are exported into the .gdx file.

Parameters:name (str) – domain or set name Returns:an ordered list of record key lists

update(self, updating_settings)

Updates the settings by merging with another one.

All domains, sets and records that are in both settings (common) or in updating_settings (new) are retained. Common elements are ordered the same way they were ordered in the original settings. New elements are appended to the common ones in the order they were in updating_settings

Parameters:updating_settings (Settings) – settings to merge with

static _update_names(names, metadatas, updating_names, updating_metadatas)

Updates a list of domain/set names and exportable flags based on reference names and flags.

to_dict(self)

Serializes the Settings object to a dict.

static from_dict(dictionary)

Deserializes Settings from a dict.

class spinetoolbox.spine_io.exporters.gdx.ExportFlag

Bases: enum.Enum

Options for exporting Set objects.

EXPORTABLE

User has declared that the set should be exported.

NON_EXPORTABLE

User has declared that the set should not be exported.

FORCED_EXPORTABLE

Set must be exported no matter what.

FORCED_NON_EXPORTABLE

Set must never be exported.

class spinetoolbox.spine_io.exporters.gdx.SetMetadata(exportable=ExportFlag.EXPORTABLE, is_additional=False)

This class holds some additional configuration for Sets.

exportable

set’s export flag

Type:ExportFlag

is_additional

True if the domain does not exist in the database but is supplied separately.

Type:bool

Parameters:

• exportable (ExportFlag) – set’s export flag
• is_additional (bool) – True if the domain does not exist in the database but is supplied separately.

__eq__(self, other)

Returns True if other is equal to this metadata.

is_exportable(self)

Returns True if Set should be exported.

is_forced(self)

Returns True if user’s export choices should be overriden.

to_dict(self)

Serializes metadata to a dictionary.

static from_dict(metadata_dict)

Deserializes metadata from a dictionary.