semprini / pymdg Goto Github PK

View Code? Open in Web Editor NEW

21.0 7.0 10.0 4.34 MB

UML Model Driven Generation of schema, code, data and documentation.

License: Other

Python 77.97% HTML 1.66% Jinja 20.00% PowerShell 0.37%

uml mdg sparx drawio django sparx-ea openapi hasura

pymdg's Introduction

pyMDG

Overview

The problem with most model driven generation is that tools force the modeller to effectively "code up" the output representation of the artifact being generated in an output specific, non-reusable model. It always ends up being simpler just to actually code up the end result.

pyMDGs goal is to use a logical model as the source, then combine with business rules to output physical level artifacts (code, schema, documentation etc). Current version supports Sparx DB (including sqlite which is the '.qea' native file format for Sparx v16+) or diagrams.net XML.

A single logical model is rich enough to generate API schemas, DB schemas, POJOs, Python Data classes etc.

The tool parses your models into generic UML classes (see metamodel below) which are then passed to jinja2 templates for generation.

My current favorite generation recipie is Hasura for a GraphQL API and generating DB migrations via Django. See the tutorial here: HasuraTutorial and check the example config in sample_recipies/sparxdb/config-sparxdb-graphql.yaml

Quickstart and docs can be found here: readthedocs

Test

Testing (powershell):

.\test.ps1

Testing (unittest):

python -m unittest

Generate

To generate code call the generate script and pass in the recipe folder. A sample recipe folder is provided in the github repo:

python mdg-tool.py generate ./sample_recipe/drawio/config-drawio-django.yaml

Or once installed into site-packages execute:

mdg-tool generate <my/config.yaml>

See the sample_recipe configs for examples

Limitations

Most templates have a limit of single inheritance and no chained inheritance (a is a specialisation of b which is a specialisation of c). The results of this are unknown.

Sparx EA XMI (versions earlier than V16) Export Process

The UML parser expects a specific package hierarchy, please see the sample EA file.

In Sparx select the domain root node (e.g. Model/Sample )
Select the publish tab at the top
Select Publish As... from top menu
Set export type as XMI 2.1
Optionally select 'Export Diagrams', 'Generate Diagram Images' and PNG format
Export to folder where you want to generate from

Note: Sparx V16+ does not need to be exported. The parser uses native SQLite file format which is the same schema as database repositories.

Draw.io Export Process

The UML parser expects a specific package layout which mimics the Sparx hierarchy, please see the sample files.

In the web editor select Export As -> XML
Uncheck 'Compressed'

Wiki documentation upload

If your generation recipe has created a file for your wiki (Confluence) then an uploader utilitity can be used. This assumes that you have done the XMI export from Sparx EA with export diagrams and generate diagram images. To generate a confluence token please see: https://confluence.atlassian.com/cloud/api-tokens-938839638.html

python mdg/confluence.py {your email} {your confluence token} {confluence page id} {path to images} {doc filename}

Nomenclature:

This diagram shows all the features and how to model in UML

Sample model

Metamodel

This diagram shows the internal classes which are passed to the templates during generation.

Build the docs

Install sphinx

 > cd pyMDG
 > sphinx-apidoc -o docs\source mdg
 > cd docs
 > make html

pymdg's People

Contributors

Stargazers

Watchers

Forkers

bionstt westpac-paul standardgalactic cuchulainx sleepergithub bellyfat skoobeeb pragyanetic20 rajamaniv softwareengineer-imerjr

pymdg's Issues

Updated open API template

---
openapi: "3.0.0"
info:
  version: 1.0.0
  title: {{ package.documentation }}
servers:
- url: http://127.0.0.1:8000/{env}/api
  variables:
    env:
      enum:
        - dev
        - test
        - prod
      default: dev
paths:{% for cls in package.classes %}{% if cls.is_abstract == False and cls.id_attribute %}
  "{{ package.path | snakecase }}{{ cls.get_name() | snakecase }}/":
    get:
      operationId: {{ package.path[1:] | snakecase | replace('/', '_') }}{{ cls.get_name() | snakecase }}_list
      parameters:
        - in: query
          name: page_key
          schema:
            type: integer
        - in: query
          name: limit
          description: 'The max number of records to return'
          schema:
            type: integer{% for assoc in cls.associations_from %}{% if assoc.cardinality.name in ["MANY_TO_ONE", "ONE_TO_ONE"] %}
        - in: query
          name: {{ assoc.destination_name | snakecase }}
          schema:
            type: {{ assoc.destination.id_attribute.dest_type }}{% endif %}{% endfor %}
      description: |
        {{ cls.documentation.replace("\n","\n        ") }}
      responses:
        200:
          description: ''
          content:
            application/json:
              schema:
                "$ref": "#/components/schemas/{{ cls.get_name() | camelcase }}_list"

  "{{ package.path | snakecase }}{{ cls.get_name() | snakecase }}/{% raw %}{{% endraw %}{{ cls.id_attribute.name | snakecase }}{% raw %}}{% endraw %}/":
    get:
      operationId: {{ package.path[1:] | snakecase | replace('/', '_') }}{{ cls.get_name() | snakecase }}_entity
      parameters:
        - name: {{ cls.id_attribute.name | snakecase }}
          in: path
          required: true
          schema:
            type: {{ cls.id_attribute.dest_type }}
      description: {{ cls.documentation.replace("\n","") }}
      responses:
        200:
          description: ''
          content:
            application/json:
              schema:
                "$ref": "#/components/schemas/{{ cls.get_name() | camelcase }}"{% if cls.associations_from != [] or cls.associations_to != [] %}
          links:{% for assoc in cls.associations_from %}{% if assoc.destination.is_abstract == False %}
            {{ assoc.destination_name | snakecase }}:
              operationId: {{ assoc.destination.package.path[1:] | snakecase | replace('/', '_') }}{{ assoc.destination.name | snakecase }}_entity
              parameters:
                {{ assoc.destination.id_attribute.name | snakecase }}: '$response.body#/{{ assoc.destination_name | snakecase }}'{% endif %}{% endfor %}{% for assoc in cls.associations_to %}{% if assoc.destination.is_abstract == False %}
            {{ assoc.source_name | snakecase }}:
              operationId: {{ assoc.source.package.path[1:] | snakecase | replace('/', '_') }}{{ assoc.source.name | snakecase }}_list
              parameters:
                {{ assoc.destination_name | snakecase }}: $request.path.{{ cls.id_attribute.name | snakecase }}{% endif %}{% endfor %}{% endif %}
{% endif %}{% endfor %}
components:
  schemas:{% for cls in package.classes %}
    {{ cls.get_name() | camelcase }}_list:
      type: object
      properties:
        result:
          type: array
          items:
            $ref: "#/components/schemas/{{ cls.get_name() | camelcase }}"
        next_page_key:
          type: string

    {{ cls.get_name() | camelcase }}_basic:
      type: object
      properties:{% if cls.id_attribute %}
        object_url:
          type: string
          format: uri
          description: "The URL to this object"
          readOnly: true
        object_type:
          type: string
          description: "The name of this object type"
          readOnly: true{% endif %}{% for attr in cls.attributes %}
        {{ attr.name | snakecase }}:
          type: {% if attr.dest_type in ["date","date-time","datetime","date time"] %}string
          format: date-time{% elif attr.classification %}string
          enum:{% for enum in attr.classification.values %}
          - {{ enum }}{% endfor %}{% elif attr.dest_type == "float" %}number
          format: float{% else %}{{ attr.dest_type }}{% endif %}{% endfor %}


    {{ cls.get_name() | camelcase }}:
      {% if not cls.supertype %}type: object
      properties:
        object_url:
          type: string
          format: uri
          description: "The URL to this object"
          readOnly: true
        object_type:
          type: string
          description: "The name of this object type"
          readOnly: true{% for attr in cls.attributes %}
        {{ attr.name | snakecase }}:
          type: {% if attr.dest_type in ["date","date-time","datetime","date time"] %}string
          format: date-time{% elif attr.classification %}string
          enum:{% for enum in attr.classification.values %}
          - {{ enum }}{% endfor %}{% elif attr.dest_type == "float" %}number
          format: float{% else %}{{ attr.dest_type }}{% endif %}{% endfor %}{% for assoc in cls.associations_from %}
        {{ assoc.destination_name | snakecase }}:
          {% if assoc.cardinality.name not in [ "MANY_TO_MANY", "ONE_TO_MANY" ] %}{% if assoc.destination.name != cls.get_name() %}{% if assoc.destination.package.name == cls.package.name %}$ref: "#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% else %}$ref: "./{{ assoc.destination.package.name | camelcase }}.yaml#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% endif %}{% else %}  type: {{ assoc.destination.id_attribute.dest_type }}{% endif %}{% else %}type: array
          items:
            oneOf:
              - type: {{ assoc.destination.id_attribute.dest_type }}
                format: uri
              - {% if assoc.destination.package.name == cls.package.name %}$ref: "#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% else %}$ref: "./{{ assoc.destination.package.name | camelcase }}.yaml#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% endif %}{% endif %}{% endfor %}{% for assoc in cls.associations_to %}{% if assoc.association_type.name == "COMPOSITION" %}
        {{ assoc.source_name | snakecase }}:
          {% if assoc.cardinality.name == "MANY_TO_ONE" %}type: array
          items:
            {% endif %}$ref: "#/components/schemas/{{ assoc.source.name | camelcase }}_basic"{% endif %}{% endfor %}{% else %}allOf:
        - $ref: "#/components/schemas/{{ cls.supertype.name | camelcase }}"
        - type: object{% if cls.attributes|length !=0 %}
          properties:{% endif %}{% for attr in cls.attributes %}
            {{ attr.name | snakecase }}:
              type: {% if attr.dest_type in ["date","date-time","datetime","date time"] %}string
              format: date-time{% elif attr.classification %}string
              enum:{% for enum in attr.classification.values %}
              - {{ enum }}{% endfor %}{% elif attr.dest_type == "float" %}number
              format: float{% else %}{{ attr.dest_type }}{% endif %}{% endfor %}{% for assoc in cls.associations_from %}
            {{ assoc.destination_name | snakecase }}:
              {% if assoc.cardinality.name not in [ "MANY_TO_MANY", "ONE_TO_MANY" ] %}{% if assoc.destination.name != cls.get_name() %}{% if assoc.destination.package.name == cls.package.name %}$ref: "#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% else %}$ref: "./{{ assoc.destination.package.name | camelcase }}.yaml#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% endif %}{% else %}  type: {{ assoc.destination.id_attribute.dest_type }}{% endif %}{% else %}type: array
              items:
                oneOf:
                  - type: {{ assoc.destination.id_attribute.dest_type }}
                    format: uri
                  - {% if assoc.destination.package.name == cls.package.name %}$ref: "#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% else %}$ref: "./{{ assoc.destination.package.name | camelcase }}.yaml#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% endif %}{% endif %}{% endfor %}{% endif %}
{% endfor %}

Sparx parse add exception handling if attribute type is None

Traceback (most recent call last):
File "c:\program files\python38\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\program files\python38\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Dev\schemagen\venv\Scripts\mdg-tool.exe_main.py", line 7, in
File "c:\dev\schemagen\venv\lib\site-packages\mdg\tools\mdg_tool.py", line 60, in main
func(args)
File "c:\dev\schemagen\venv\lib\site-packages\mdg\tools\mdg_tool.py", line 22, in generate
generate()
File "c:\dev\schemagen\venv\lib\site-packages\mdg\generate.py", line 19, in generate
model_package, test_cases = parse()
File "c:\dev\schemagen\venv\lib\site-packages\mdg\parse.py", line 36, in parse
model_package, test_cases = parser()
File "c:\dev\schemagen\venv\lib\site-packages\mdg\sparx_xmi\parse.py", line 51, in parse_uml
model_package = package_parse(model_element, tree, None)
File "c:\dev\schemagen\venv\lib\site-packages\mdg\sparx_xmi\parse.py", line 134, in package_parse
package_parse_children(element, package)
File "c:\dev\schemagen\venv\lib\site-packages\mdg\sparx_xmi\parse.py", line 145, in package_parse_children
pkg = package_parse(child, package.root_element, package)
File "c:\dev\schemagen\venv\lib\site-packages\mdg\sparx_xmi\parse.py", line 134, in package_parse
package_parse_children(element, package)
File "c:\dev\schemagen\venv\lib\site-packages\mdg\sparx_xmi\parse.py", line 149, in package_parse_children
cls = class_parse(package, child, package.root_element)
File "c:\dev\schemagen\venv\lib\site-packages\mdg\sparx_xmi\parse.py", line 401, in class_parse
attr = attr_parse(cls, child, root)
File "c:\dev\schemagen\venv\lib\site-packages\mdg\sparx_xmi\parse.py", line 432, in attr_parse
type_id = type_elem.get('{%s}idref' % ns['xmi'])
AttributeError: 'NoneType' object has no attribute 'get'

Sparx Enterprise Architect parsing XMI - more detailed instructions

Hello, it would be great, if you can explain on a quick tutorial site, which commands needs to be used e.g. for Enterprise Architect XMI Export and what yml is doing what and example commands like this one:

C:\Users\Jannis\Downloads\pyMDG-master\pyMDG-master>python mdg-tool.py generate ./sample_recipes/sparx/config-sparx-python.yaml

The YML then contains data about the test.xmi etc.

Tutorial incorrectly says 2 relation types.

Aggregation and Composition need to be documented

snakecase does not handle multiple caps

source: EFTPOS
Result: e_f_t_p_o_s
Expected: eftpos

Add version to SparxDB parse

TPackage.version
def package_parse(session, tpackage: TPackage, parent_package: Optional[UMLPackage]):

    package = UMLPackage(tpackage.package_id, tpackage.name, parent_package)

Snakecase fails on null

Traceback (most recent call last):
File "c:\program files\python38\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\program files\python38\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Dev\python\venv\Scripts\mdg_generate.exe_main.py", line 7, in
File "c:\dev\python\venv\lib\site-packages\mdg\tools\mdg_generate.py", line 18, in main
generate()
File "c:\dev\python\venv\lib\site-packages\mdg\generate.py", line 26, in generate
output_model(model_package)
File "c:\dev\python\venv\lib\site-packages\mdg\render.py", line 159, in output_model
output_model(child)
File "c:\dev\python\venv\lib\site-packages\mdg\render.py", line 143, in output_model
output_level_package(source_template, dest_file_template, package)
File "c:\dev\python\venv\lib\site-packages\mdg\render.py", line 45, in output_level_package
fh.write(source_template.render(package=package))
File "c:\dev\python\venv\lib\site-packages\jinja2\environment.py", line 1090, in render
self.environment.handle_exception()
File "c:\dev\python\venv\lib\site-packages\jinja2\environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "c:\dev\python\venv\lib\site-packages\jinja2_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "./sample_recipie/templates\openapi.yaml.jinja", line 28, in top-level template code
operationId: {{ cls.path | snakecase }}list
File "c:\dev\python\venv\lib\site-packages\mdg\tools\case.py", line 150, in snakecase
val = re.sub(r"[A-Z]", lambda matched: '' + lowercase(matched.group(0)), string)
File "c:\program files\python38\lib\re.py", line 210, in sub
return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

Generation type should be configurable per output

generation_type: default

model_templates:

dest: "avro/{{cls.package.name | snakecase }}_{{ cls.name | snakecase }}.avsc"
level: class
source: "Schema/avro.avsc.jinja"
generation_type: avro

openapi template

openapi: "3.0.0"
info:
version: 1.0.0
title: {{ package.documentation }}
servers:

url: http://127.0.0.1:8000/{env}/api
variables:
env:
enum:
- dev
- test
- prod
default: dev
paths:{% for cls in package.classes %}{% if cls.is_abstract == False and cls.id_attribute %}
"{{ package.path | snakecase }}{{ cls.get_name() | snakecase }}/":
get:
operationId: {{ package.path[1:] | snakecase | replace('/', '_') }}{{ cls.get_name() | snakecase }}_list
parameters:
- in: query
name: page_key
schema:
type: integer
- in: query
name: limit
description: 'The max number of records to return'
schema:
type: integer{% for assoc in cls.associations_from %}{% if assoc.cardinality.name in ["MANY_TO_ONE", "ONE_TO_ONE"] %}
- in: query
name: {{ assoc.destination_name | snakecase }}
schema:
type: {{ assoc.destination.id_attribute.dest_type }}{% endif %}{% endfor %}
description: ''
responses:
200:
description: ''
content:
application/json:
schema:
"$ref": "#/components/schemas/{{ cls.get_name() | camelcase }}_list"

"{{ package.path | snakecase }}{{ cls.get_name() | snakecase }}/{% raw %}{{% endraw %}{{ cls.id_attribute.name | snakecase }}{% raw %}}{% endraw %}/":
get:
operationId: {{ package.path[1:] | snakecase | replace('/', '_') }}{{ cls.get_name() | snakecase }}entity
parameters:
- name: {{ cls.id_attribute.name | snakecase }}
in: path
required: true
schema:
type: {{ cls.id_attribute.dest_type }}
description: ''
responses:
200:
description: ''
content:
application/json:
schema:
"$ref": "#/components/schemas/{{ cls.get_name() | camelcase }}"{% if cls.associations_from != [] %}
links:{% for assoc in cls.associations_from %}{% if assoc.destination.is_abstract == False %}
{{ assoc.destination_name | snakecase }}:
operationId: {{ assoc.destination.package.path[1:] | snakecase | replace('/', '') }}{{ assoc.destination.name | snakecase }}_entity
parameters:
{{ assoc.destination.id_attribute.name | snakecase }}: '$response.body#/{{ assoc.destination_name | snakecase }}'{% endif %}{% endfor %}{% endif %}
{% endif %}{% endfor %}
components:
schemas:{% for cls in package.classes %}
{{ cls.get_name() | camelcase }}_list:
type: object
properties:
result:
type: array
items:
$ref: "#/components/schemas/{{ cls.get_name() | camelcase }}"
next_page_key:
type: string

{{ cls.get_name() | camelcase }}_basic:
type: object
properties:{% for attr in cls.attributes %}
{{ attr.name | snakecase }}:
type: {% if attr.dest_type in ["date","date-time","datetime","date time"] %}string
format: date-time{% elif attr.classification %}string
enum:{% for enum in attr.classification.values %}
- {{ enum }}{% endfor %}{% else %}{{ attr.dest_type }}{% endif %}{% endfor %}

{{ cls.get_name() | camelcase }}:
{% if not cls.supertype %}type: object
properties:
object_url:
type: string
format: uri
description: "The URL to this object"
readOnly: true
object_type:
type: string
description: "The name of this object type"
readOnly: true{% for attr in cls.attributes %}
{{ attr.name | snakecase }}:
type: {% if attr.dest_type in ["date","date-time","datetime","date time"] %}string
format: date-time{% elif attr.classification %}string
enum:{% for enum in attr.classification.values %}
- {{ enum }}{% endfor %}{% else %}{{ attr.dest_type }}{% endif %}{% endfor %}{% for assoc in cls.associations_from %}
{{ assoc.destination_name | snakecase }}:
{% if assoc.cardinality.name not in [ "MANY_TO_MANY", "ONE_TO_MANY" ] %}{% if assoc.destination.name != cls.get_name() %}{% if assoc.destination.package.name == cls.package.name %}$ref: "#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% else %}$ref: "./{{ assoc.destination.package.name | camelcase }}.yaml#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% endif %}{% else %} type: {{ assoc.destination.id_attribute.dest_type }}{% endif %}{% else %}type: array
items:
oneOf:
- type: {{ assoc.destination.id_attribute.dest_type }}
format: uri
- {% if assoc.destination.package.name == cls.package.name %}$ref: "#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% else %}$ref: "./{{ assoc.destination.package.name | camelcase }}.yaml#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% endif %}{% endif %}{% endfor %}{% for assoc in cls.associations_to %}{% if assoc.association_type.name == "COMPOSITION" %}
{{ assoc.source_name | snakecase }}:
{% if assoc.cardinality.name == "MANY_TO_ONE" %}type: array
items:
{% endif %}$ref: "#/components/schemas/{{ assoc.source.name | camelcase }}_basic"{% endif %}{% endfor %}{% else %}allOf:
- $ref: "#/components/schemas/{{ cls.supertype.name | camelcase }}"
- type: object{% if cls.attributes|length !=0 %}
properties:{% endif %}{% for attr in cls.attributes %}
{{ attr.name | snakecase }}:
type: {% if attr.dest_type in ["date","date-time","datetime","date time"] %}string
format: date-time{% elif attr.classification %}string
enum:{% for enum in attr.classification.values %}
- {{ enum }}{% endfor %}{% else %}{{ attr.dest_type }}{% endif %}{% endfor %}{% for assoc in cls.associations_from %}
{{ assoc.destination_name | snakecase }}:
{% if assoc.cardinality.name not in [ "MANY_TO_MANY", "ONE_TO_MANY" ] %}{% if assoc.destination.name != cls.get_name() %}{% if assoc.destination.package.name == cls.package.name %}$ref: "#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% else %}$ref: "./{{ assoc.destination.package.name | camelcase }}.yaml#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% endif %}{% else %} type: {{ assoc.destination.id_attribute.dest_type }}{% endif %}{% else %}type: array
items:
oneOf:
- type: {{ assoc.destination.id_attribute.dest_type }}
format: uri
- {% if assoc.destination.package.name == cls.package.name %}$ref: "#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% else %}$ref: "./{{ assoc.destination.package.name | camelcase }}.yaml#/components/schemas/{{ assoc.destination.name | camelcase }}_basic"{% endif %}{% endif %}{% endfor %}{% endif %}
{% endfor %}

Add parsing for attribute documentation

Sparx local DBs can have the same id for packages as classes

    def find_by_id(self, id: Union[int, str], find_type: Optional[str]=None):
        """ Finds UMLPackage, UMLClass, UMLEnumeration or UMLInstance object with specified Id
        Looks for classes part of this package and all sub-packages
        """
        if str(self.id) == str(id) and find_type in (None, "package"):
            return self

        for cls in self.classes:
            if str(cls.id) == str(id) and find_type in (None, "class"):
                return cls

        for ins in self.instances:
            if str(ins.id) == str(id) and find_type in (None, "instance"):
                return ins

        for enum in self.enumerations:
            if str(enum.id) == str(id) and find_type in (None, "enumeration"):
                return enum

        for child in self.children:
            res = child.find_by_id(id, find_type)
            if res is not None:
                return res

Sparx DB parse not classifying attributes with enumerations

            attr.classification = package.root_package.find_by_id(attr.classification_id, 'enumeration')

Draw.io Parse not stripping class name correctly

        <object label="&lt;b&gt;ANZSIC_Division&lt;br&gt;&lt;/b&gt;" UMLType="Class" id="RbtesccJiY1XfNZTxO5F-41">
          <mxCell style="swimlane;html=1;fontStyle=0;childLayout=stackLayout;horizontal=1;startSize=30;fillColor=#ffe6cc;horizontalStack=0;resizeParent=1;resizeLast=0;collapsible=1;marginBottom=0;swimlaneFillColor=#ffffff;shadow=0;comic=0;labelBackgroundColor=none;strokeWidth=1;fontFamily=Verdana;fontSize=10;align=center;strokeColor=#d79b00;arcSize=10;rounded=1;" parent="RbtesccJiY1XfNZTxO5F-40" vertex="1">
            <mxGeometry x="40" y="60" width="170" height="110" as="geometry">
              <mxRectangle x="260" y="80" width="100" height="50" as="alternateBounds" />
            </mxGeometry>
          </mxCell>
        </object>

OSError: [Errno 22] Invalid argument: 'c:\\Dev\\DataProduct\\build\\avro\\Reference.ANZSIC_Division<br.avsc'

Sparxdb parse associations should specify class in find by id

158: find_by_id(connector.end_object_id, "class")

Abstract classes do not need Id

2021-02-15 14:18:47,926 | mdg.parse | INFO | Base Model Package: Common
Validation Errors:
Class error: /Common/Base/ManagedEntity | no primary key

Need sorting of composed classes

uml py

class UMLClass(object):
def init(self, package: UMLPackage, name: str, id: Union[int, str]):
...
self.composed_of: List[UMLClass] = []

parse py
54:
package_sort_classes(model_package)
256:
def package_sort_classes(package):
unordered_list = package.classes
unordered_composed_list = []
for cls in unordered_list:
if cls.composed_of != []:
unordered_composed_list.append(cls)
package.classes.remove(cls)

# This is crap, do some propper sorting
unordered_list = unordered_composed_list
composed_list = []
for cls1 in unordered_list:
    for cls2 in cls1.composed_of:
        if cls2 in unordered_composed_list:
            unordered_composed_list.remove(cls2)
            composed_list.append(cls2)
            
composed_list += unordered_composed_list 
package.classes += composed_list

for child in package.children:
    package_sort_classes(child)

serializers py jinja
from rest_framework import serializers
from drf_nest.serializer_fields import TypeField

{% for cls in package.classes %}
from .models import {{ cls.name | case_class }}{% endfor %}

{% for cls in package.classes %}
class {{ cls.name | case_class }}Serializer(serializers.HyperlinkedModelSerializer):
type = TypeField()

{% for assoc in cls.associations_to %}{% if assoc.association_type.name == "COMPOSITION" %}
{{ assoc.source_name | snakecase }} = {{ assoc.source.name | case_class }}Serializer(required=False{% if assoc.cardinality.name == "MANY_TO_ONE" %}, many=True{% endif %}){% endif %}{% endfor %}

class Meta:
    model = {{ cls.name | case_class }}
    fields = ('type', 'url', 
                {% for attr in cls.attributes %}'{{ attr.name | snakecase }}',{% endfor %}{% if cls.generalization != None %}{% for attr in cls.generalization.attributes %}'{{ attr.name | snakecase }}',{% endfor %}{% endif %}
				{% for assoc in cls.associations_from %}'{{ assoc.destination_name | snakecase }}',{% endfor %}
                {% for assoc in cls.associations_to %}'{{ assoc.source_name | snakecase }}',{% endfor %}
            )

{% endfor %}

models py jinja
from django.utils.translation import gettext_lazy as _
from django.db import models
from simple_history.models import HistoricalRecords

from {{ package.root_package.name | snakecase }}.validators import validate_even

{% for cls in package.classes %}{% if cls.generalization != None %}{% if cls.generalization.package != package %}
from {{ cls.generalization.package.name | case_package }}.models import {{ cls.generalization.name | case_class }}{% endif %}{% endif %}{% endfor %}

{% for enum in package.enumerations %}class ENUM_{{ enum.name | case_class }}(models.TextChoices):
{% for attr in enum.values %}
{{ attr.replace('-','').replace('/','') | snakecase | uppercase }} = '{{ attr }}', _('{{ attr }}'){% endfor %}

{% endfor %}
{% for cls in package.classes %}{% if cls.is_abstract %}class {{ cls.name | case_class }}( models.Model ):
{% for attr in cls.attributes %}
{% if attr.classification %} {{ attr.name | snakecase }} = models.CharField( max_length=100, choices=ENUM_{{ attr.classification.name | case_class }}.choices, blank=True, null=True )
{% else %} {{ attr.name | snakecase }} = models.{% if attr.stereotype == "auto" %}AutoField{% else %}{{ attr.dest_type }}{% endif %}( {% if attr.is_id %}primary_key=True, {% else %}blank=True, null=True, {% endif %}{% if attr.length %}max_length={{ attr.length }}{% endif %} )
{% endif %}{% endfor %}
{% if cls.is_abstract %} class Meta:
abstract = True
{% endif %}{% endif %}
{% endfor %}

{% for cls in package.classes %}{% if not cls.is_abstract %}class {{ cls.name | case_class }}( {% if cls.generalization %}{{ cls.generalization.name | case_class }}{% else %}models.Model{% endif %} ):
{% for attr in cls.attributes %}
{% if attr.classification %} {{ attr.name | snakecase }} = models.CharField( max_length=100, choices=ENUM_{{ attr.classification.name | case_class }}.choices, blank=True, null=True )
{% else %} {{ attr.name | snakecase }} = models.{% if attr.stereotype == "auto" %}AutoField{% else %}{{ attr.dest_type }}{% endif %}( {% if attr.is_id %}primary_key=True, {% else %}blank=True, null=True, {% endif %}{% if attr.dest_type == "DecimalField" %}max_digits=10, decimal_places=2, {% endif %}{% if attr.length %}max_length={{ attr.length }}{% endif %}{% if attr.validations != [] %}validators=[validate_even]{% endif %} )
{% endif %}{% endfor %}

{% for rel in cls.associations_from %}{% if rel.association_type.name == "ASSOCIATION" %}
{% if rel.cardinality.name == "ONE_TO_ONE" %}{{ rel.destination_name | snakecase }} = models.OneToOneField( '{{ rel.destination.package.name | case_package }}.{{ rel.destination.name | case_class }}', on_delete=models.CASCADE, blank=True, null=True )
{% elif rel.cardinality.name == "ONE_TO_MANY" %}{{ rel.destination_name | snakecase }} = models.ForeignKey( '{{ rel.destination.package.name | case_package }}.{{ rel.destination.name | case_class }}', related_name='{{ rel.destination_name | snakecase }}', on_delete=models.CASCADE, blank=True, null=True )
{% elif rel.cardinality.name == "MANY_TO_MANY" %}{{ rel.destination_name | snakecase }} = models.ManyToManyField( '{{ rel.destination.package.name | case_package }}.{{ rel.destination.name | case_class }}', blank=True ){% endif %}
{% elif rel.association_type.name == "COMPOSITION" %}
{% if rel.cardinality.name == "ONE_TO_ONE" %}{{ rel.destination_name | snakecase }} = models.OneToOneField( '{{ rel.destination.package.name | case_package }}.{{ rel.destination.name | case_class }}', related_name='{{ rel.source_name | snakecase }}', on_delete=models.CASCADE )
{% else %}{{ rel.destination_name | snakecase }} = models.ForeignKey( '{{ rel.destination.package.name | case_package }}.{{ rel.destination.name | case_class }}', related_name='{{ rel.source_name | snakecase }}', on_delete=models.CASCADE ){% endif %}
{% endif %}{% endfor %}
{% for rel in cls.associations_to %}{% if rel.association_type.name != "COMPOSITION" %}
{% if rel.cardinality.name == "MANY_TO_ONE" %}{{ rel.source_name | snakecase }} = models.ForeignKey( '{{ rel.source.package.name | case_package }}.{{ rel.source.name | case_class }}', on_delete=models.CASCADE, related_name='{{ rel.destination_name | snakecase }}', blank=True, null=True ){% endif %}
{% endif %}{% endfor %}

{% if 'auditable' in cls.stereotypes %}
history = HistoricalRecords(){% endif%}

{% if cls.is_abstract %} class Meta:
abstract = True
{% endif %}{% endif %}
{% endfor %}

Add get_name to UML class and attribute

def get_name(self) -> str:
    if self.alias:
        return f"{self.alias}"
    else:
        return f"{self.name}"

Add a daemon mode to check for package version changes and run automatically

Simple script to poll version in Sparx TPackage table and run configured generation via mdg-tool generate

Add association type to sparx parse

Supporting
ASSOCIATION = 1
COMPOSITION = 2
&
AGGREGATION = 3

Add support for Sparx DB repos

As per V16 of Sparx EA, the file and DB repos use the same schema. The file repos use SQLite.

UMLPackage path root incorrect

uml/init.py
if self.parent is None:
self.root_package = self
self.path = '/' + self.name + '/'

also in mdg\parse_init_.py

Add ddl generation_fields

"ddl":{
    "Integer": "NUMBER",
    "int": "NUMBER",
    "String": "VARCHAR2",
    "string": "VARCHAR2",
    "str": "VARCHAR2",
    "decimal": "NUMBER",
},

}

Unable to create self references

2021-03-09 12:45:14,143 | mdg.sparx_xmi.parse | WARNING | Unable to create association id=EAID_512B42F1_B8E8_4516_8912_54FECA691841

association source & dest is same object. Cardinality = many to one

Should check if find returns none

pyMDG/mdg/parse/sparx_db.py

Line 158 in e3986e8

dest = package.root_package.find_by_id(connector.end_object_id, 'class')

Sparx association naming source & dest swapped

Since association defaults to object name, if a name is specified the source name should be applied to the destination object.

    # If it's an association to or from a multiple then pluralize the name
    if dest_element.get('name') is not None:
        association.source_name = dest_element.get('name')
    else:
...
    if source_element.get('name') is not None:
        association.destination_name = source_element.get('name')
    else:
...```

Check for eval to "" in snakecase

def snakecase(string):
if string is None or f"{string}" == "":
return string

avro generation fields

"avro": {
    "bool": "boolean",
    "Date": "date",
    "dateTime": "date-time",
    "decimal": "number",
    "Decimal": "number",
    "enum": "enum",
    "int": "int",
    "integer": "int",
    "bigint": "int",
    "Integer": "int",
    "long": "int",
    "String": "string",
    "str": "string",
},

Django template should allow custom signals

app.py

        try:
            from Sync.signals import init_signals
            init_signals()
        except ImportError:
            print("No Custom Signals")

Map of python field types

"python": {
    "Integer": "int",
    "String": "str",
    "string": "str",
    "Float": "float",
    "Numeric": "Decimal",
    "Boolean": "bool",
    "DECIMAL": "Decimal",
    "CHAR": "str",
    "TIMESTAMP": "datetime.timestamp",
    "DATE": "date",
    "INTEGER": "int",
    "VARCHAR": "str",
},

UML json should convert to JSONField

json = models.CharField( blank=True, null=True, max_length=100 )

json = models.JSONField( blank=True, null=True )

Specify verbose logging on cmd line

Add a -v to mdg-tool options to set logging level to debug.

Sample generation fails on root template

Generating model output for package /SampleIndustry/TestDomain/
Traceback (most recent call last):
File "c:\users\atkin\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main
return run_code(code, main_globals, None,
exec(code, run_globals)
File "C:\dev\python\venv3\Scripts\mdg_generate.exe_main.py", line 7, in
File "c:\dev\python\venv3\lib\site-packages\mdg\tools\mdg_generate.py", line 18, in main
generate()
File "c:\dev\python\venv3\lib\site-packages\mdg\generate.py", line 26, in generate
output_model(model_package)
File "c:\dev\python\venv3\lib\site-packages\mdg\render.py", line 158, in output_model
output_model(child)
File "c:\dev\python\venv3\lib\site-packages\mdg\render.py", line 153, in output_model
raise ValueError("'{}' is not a valid template level".format(template_definition['level']))
ValueError: 'root' is not a valid template level

Circular reference inside model_package after parsing

Hello,
the root_package inside the model_package is not a deep copy.
So its causing a circular reference (I can open it over and over again).
Is this wanted by you?
I checked the model_package content during debugging inside the /generate/init.py file

Add status to class in XMI parse

<element xmi:idref=... xmi:type="uml:Class" name="History" scope="public">
    <model package=... tpos="0" ea_localid="15" ea_eleType="element"/>
    <properties isSpecification="false" sType="Class" nType="0" scope="public" isRoot="false" isLeaf="false" isAbstract="false" isActive="false"/>
    <project author="M809509" version="1.0" phase="1.0" created="2021-02-02 13:44:42" modified="2021-02-02 13:55:05" complexity="1" 
        status="Proposed"/>

parse.py

    detail = root.xpath("//element[@xmi:idref='%s']" % cls.id, namespaces=ns)[0]
    project = detail.find('project')
    cls.status= properties.get('status')

Snakecase should simply use lower if underscores are in name

Expected
ABC_DEF -> abc_def

Actual
ABC_DEF -> A_B_C__D_E_F_

Sparx Reversed explicit relationship names

# If it's an association to or from a multiple then pluralize the name
# TODO: Allow pluralized name to be specified in UML
# Use opposing ends class name as attribute name for association
association.destination_name = association.destination.name.lower()
if association.destination_multiplicity[1] == '*':
    association.destination_name += 's'
association.source_name = association.source.name.lower()
if association.source_multiplicity[1] == '*':
    association.source_name += 's'

if dest_element.get('name') is not None:
    association.source_name = dest_element.get('name')

if source_element.get('name') is not None:
    association.destination_name = source_element.get('name')

# print('Assoc in {}: {} to {}: type = {}'.format(self.source.name, self.source_name, self.destination_name,
# self.association_type) )
return association

Add support for BOUML

Hi, i like your project.
Sparx EA is great but for small company or freelancer the price is not affordable.
I think it would be great if pyMDG supports open source modeling software. BOUML is really great https://www.bouml.fr/
I checked the code and i see in parser, you already left a place to add custom parser.
Thanks

snakecase does not handle jinja2.runtime.Undefined

Should be:

def snakecase(string):
    if string is None or f"{string}" == "":

Test generation should output numbers as natural

With Amount attribute defined as number (integer, float, decimal):

Expected:

{
  "Amount": 10
...
}

Actual:

{
  "Amount": "10"
 ...
}

uml django should convert datetime to DateTimeField

update_time = models.datetime( blank=True, null=True, )

to:

update_time = models.DateTimeField( blank=True, null=True, )

AttributeError: module 'mdg.generate' has no attribute 'main'

On mdg_generate

Traceback (most recent call last):
File "C:\Dev\DataProduct\venv\Scripts\mdg_generate-script.py", line 33, in
sys.exit(load_entry_point('pymdg==0.3a1', 'console_scripts', 'mdg_generate')())
File "C:\Dev\DataProduct\venv\Scripts\mdg_generate-script.py", line 25, in importlib_load_entry_point
return next(matches).load()
File "c:\program files\python38\lib\importlib\metadata.py", line 79, in load
return functools.reduce(getattr, attrs, module)
AttributeError: module 'mdg.generate' has no attribute 'main'

attribute id fields set incorrectly in sparx_db parse

if txref is not None:
    attr.is_id = bool(re.findall('@NAME=isID@ENDNAME;@TYPE=Boolean@ENDTYPE;@VALU=1@ENDVALU;', txref.description))
    # print(f"{attr.name} | {attr.is_id} | {txref.description}")
    if attr.is_id:
        attr.parent.id_attribute = attr

django many to one & one to many reversed

{% for rel in cls.associations_from %}{% if rel.association_type.name == "ASSOCIATION" %}
{% if rel.cardinality.name == "ONE_TO_ONE" %}{{ rel.destination_name | snakecase }} = models.OneToOneField( '{{ rel.destination.package.name | case_package }}.{{ rel.destination.name | case_class }}', on_delete=models.CASCADE, blank=True, null=True )
{% elif rel.cardinality.name == "MANY_TO_ONE" %}{{ rel.destination_name | snakecase }} = models.ForeignKey( '{{ rel.destination.package.name | case_package }}.{{ rel.destination.name | case_class }}', related_name='{{ rel.source_name | snakecase }}', on_delete=models.CASCADE, blank=True, null=True )
{% elif rel.cardinality.name == "MANY_TO_MANY" %}{{ rel.destination_name | snakecase }} = models.ManyToManyField( '{{ rel.destination.package.name | case_package }}.{{ rel.destination.name | case_class }}', blank=True ){% endif %}
{% elif rel.association_type.name == "COMPOSITION" %}
{% if rel.cardinality.name == "ONE_TO_ONE" %}{{ rel.destination_name | snakecase }} = models.OneToOneField( '{{ rel.destination.package.name | case_package }}.{{ rel.destination.name | case_class }}', related_name='{{ rel.source_name | snakecase }}', on_delete=models.CASCADE )
{% else %}{{ rel.destination_name | snakecase }} = models.ForeignKey( '{{ rel.destination.package.name | case_package }}.{{ rel.destination.name | case_class }}', related_name='{{ rel.source_name | snakecase }}', on_delete=models.CASCADE ){% endif %}
{% endif %}{% endfor %}
{% for rel in cls.associations_to %}{% if rel.association_type.name != "COMPOSITION" %}
{% if rel.cardinality.name == "ONE_TO_MANY" %}{{ rel.source_name | snakecase }} = models.ForeignKey( '{{ rel.source.package.name | case_package }}.{{ rel.source.name | case_class }}', on_delete=models.CASCADE, related_name='{{ rel.destination_name | snakecase }}', blank=True, null=True ){% endif %}
{% endif %}{% endfor %}

mdg-tool dumps on sparx XMI fails with AttributeError: 'tuple' object has no attribute 'Meta'

Works for drawio though,

mdg-tool dumps sample_recipes\config-sparx-docs.yaml
2022-08-08 10:49:54,753 | mdg.config | INFO | Config file loaded: sample_recipes\config-sparx-docs.yaml
2022-08-08 10:49:54,802 | mdg.parse.sparx_xmi | INFO | Parsing models
2022-08-08 10:49:54,825 | mdg.parse.sparx_xmi | INFO | Parsing test cases
2022-08-08 10:49:54,828 | mdg.parse | INFO | Base Model Package: SampleIndustry
Traceback (most recent call last):
File "C:\Dev\python\pyMDG-master\mdg-tool.py", line 6, in
main()
File "C:\Dev\python\pyMDG-master\mdg\tools\mdg_tool.py", line 77, in main
func(args)
File "C:\Dev\python\pyMDG-master\mdg\tools\mdg_tool.py", line 36, in dumps
print(dumps(model_package))
File "C:\Dev\python\pyMDG-master\mdg\uml_init_.py", line 368, in dumps
output = obj_to_dict(package)
File "C:\Dev\python\pyMDG-master\mdg\tools\io.py", line 38, in obj_to_dict
new_value.append(obj_to_dict(element))
File "C:\Dev\python\pyMDG-master\mdg\tools\io.py", line 38, in obj_to_dict
new_value.append(obj_to_dict(element))
File "C:\Dev\python\pyMDG-master\mdg\tools\io.py", line 28, in obj_to_dict
value = getattr(value, value.Meta.id_field)
AttributeError: 'tuple' object has no attribute 'Meta'

Inherited classes may have Id

2021-02-15 14:18:47,926 | mdg.parse | INFO | Base Model Package: Common
Validation Errors:
Class error: /Common/Involved Party/Customer | primary key in both class and supertype

Dumps missing associations

                "associations_from": ["7UrE3CHdKnYvW7Xawwn1-13", "X6_GDUgpjBKw1VpaA0fO-1"],
           "associations": [],

from and to may not be needed

{
  "transactions_state": "Active"
...
}