u9n / dlms-cosem Goto Github PK

We need to support service specific block transfer, at least for the GET service.
The service specific block transfer is different from the GeneralBlockTransfer.

The service specific GET block transfer is still contained done using GET-requests and GET-responses. But when the result is larger than negotiated PDU-size the meter will need to send data in blocks. It does this by sending a GET-response with the type GetType.WITH_BLOCK.

Upon receiving this the client should request the next block with a GET request to the same instance and attribute but setting the type to GetRequestType.NEXT

The last block sent from the meter will be a GET-response with block where the last-block attribute is true.

To be able to support this feature we should do some changes:

1. Instead of sending a GetRequest with request_type=GetRequestType.NEXT we should send a GetRequestNext apdu.
   By using new classes for the different types of request and responses we can easier track the the state of the connection and we can easier control the encoding and decoding of the bytes for each specific APDU. We should however introduce a new factoryclass that can generate the correct class when given the data of a GetRequest.
2. We need to delay interpretation of the byte sent in the request. Now with a GetRequest.NORMAL we can just parse the data since it is all there. But when we introduce the block transfer we need to wait for all the blocks to arrive before we can try to parse the data. The simplest thing would be to just supply the bytes and let the client parse data when it feels eveything is done.

New classes for the GetRequest factory:

• GetRequestNormal
• GetRequestNext
• GetRequestWithList (raise not implemented error)

New classes for the GetResponseFactory:

• GetResponseNormal
• GetResponseNormalWithError
• GetResponseWithBlock
• GetResponseLastBlock
• GetResponseLastBlockWithError
• GetResponseWithList (raise not implemented error)

Error handling:

The service specific block transfers are arguably worse in reliability than the GeneralBlockTransfer. We cannot request lost blocks. If we receive a block out of order we need to abort the whole transmission.

1. If the meter cannot produce the next block for some reasone it will respond with a GetResponseLastBlockWithError but indicate an error instead of data. This ends the transmission.
2. If the meter receives a GetRequestNext with a block number that is not the last block the meter sent, the meter will interpret this as an abortion and will abort the transmission by sending a GetResponseLastBlockWithError with an error DataAccessResult.LONG_GET_ABORTED.
3. The client might send a GetRequestNext when no block transfer has been initiated. The meter will then respond with a GetResponseLastBlockWithError with the error DataAccessResult.NO_LONG_GET_IN_PROGRESS.
4. If the client receives a block from the meter that is not the next block in the sequence the client shall abort the transmission and not send a GetRequestNext.
5. If for any reason in the cases 1-3 the meter is not able to produce a GetResponseLastBlockWithError it can instead send a GetResponseNormalWithError

We do not send a shutdown in the BlockingTCPTransport on the socket before closing and we have reason to believe that this sometimes causes meters modem to hang thinking the socket is still open.

We should add a call to shutdown so we make it easier for modems to handle the disconnect.

Right now the HDLC layer only supports messages that are within the standard window sized. (128 bytes)
We need to extend the connection and state of HDLC to allow for sending and receiving segmented frames.

Requesting support for Kamstrup Omnipower HAN-module and Aidon AMI system smart meters with "HAN-NVE" RJ45 interface.

"HAN-NVE" has a physical RJ45 port which in Sweden and Scandinavia is referred to as the "Norwegian RJ45 HAN-port standard".

https://hanporten.se/norska/porten/

https://hanporten.se/norska/protokollet/

https://hanporten.se/norska/kretsschema/

Google search says that they at least sell an Kamstrup HAN-module with "HAN-NVE" RJ45 HAN interface with part number 6840004 which according to its specifications listed in the data sheet it looks like the reader needs to comply with MBUS EN-13757-2 standard as per recommendation from NVE (Norwegian Water Resources and Energy Directorate) for Norway, a.k.a. AMS+HAN version 2.0 or IEC 62056-7-5, annex D ”MBUS” slave, e.g. an IEC 62056-21 protocol mode B client.

https://www.nek.no/wp-content/uploads/2018/10/Kamstrup-HAN-NVE-interface-description_rev_3_1.pdf

https://www.nek.no/wp-content/uploads/2017/10/AMS-HAN-personvernnotat-h%C3%B8ringsversjon.pdf

https://byggebolig.no/imageoriginals/88b3d1774ecb41e6a3fe067ae9e6a893.pdf

https://www.energiforetagen.se/globalassets/energiforetagen/det-erbjuder-vi/publikationer/branschrekommendation-lokalt-granssnitt-v1-2-2018.pdf

https://www.utomhusliv.se/wp-content/uploads/2020/10/Specifikation-f%C3%B6r-HAN-modulen-f%C3%B6r-elm%C3%A4tare-engelska.pdf

Google also says that Aidon AMI system smart meters use the same type of HAN-NVE HAN with RJ45 port with M-Bus interface:

https://www.skekraft.se/wp-content/uploads/2021/03/Aidon_Feature_description_RJ45_HAN_Interface_EN.pdf

https://www.tekniskaverken.se/siteassets/tekniska-verken/elnat/aidonfd-rj45-han-interface-se-v13a.pdf

There looks to be some guides for parsing the protocol and getting the data through some kind of M-Bus converter here:

https://www.kode24.no/guider/smart-meter-part-1-getting-the-meter-data/71287300

https://xipher.dk/posts/2020-05-17-using-esp8266-to-monitor-kamstrup-omnipower/

https://github.com/Claustn/esp8266-kamstrup-mqtt

https://github.com/roarfred/AmsToMqttBridge/blob/master/Samples/Kaifa/readme.md

https://github.com/roarfred/AmsToMqttBridge/blob/master/Samples/Kamstrup/obisdata.md

https://hanporten.se/norska/protokollet/

"The meter acts as Mbus master pushing data. To read the information, a circuit for Mbus' slave is needed."

"They are following the DLMS (Device Language Message Specification) protocol and are sent inside HDLC frames and contains OBIS (Object Identification System) codes that describe the electricity usage. Everything is part of IEC 62056 which is a set of standards for electricity metering data exchange."

with the swich_client_type it is easy to switch a public client into a management client.

But it is not doing the correct job when going from a management client to a public one.

We need to remove the encryption keys, otherwise the connection still thinks it is encrypted.

Since public and managment client is the most common one we should make the switch_client_type more general and then have separate methods for make_public and make_management to handle the standard cases ontop of switch_client_type

I'm having problems decoding data from two Kaifa meters. The exception I'm getting is "HCS is not correct Calculated".

This is data from first meter:
7ea09b01000110561be6e7000f40000000090c07e7090401103400ff800000021209074b464d5f30303109103733343031353730313132353335343409084d41333034483444060000044f0600000000060000000006000000c0060000088f06000005aa060000057c06000008da06000008f906000008e6090c07e7090401103400ff8000000608c141c9060000000006001ae03806013151959d787e
This is data from the second meter:
7ea11d01000110b0aee6e7000f4000000000022409060100000281ff09074b464d5f30303109060000600100ff09103733343031353730333037383433393509060000600107ff09074d41333034483409060100010700ff060000017209060100020700ff060000000009060100030700ff060000000009060100040700ff0600000050090601001f0700ff06000001be09060100330700ff060000047b09060100470700ff060000009409060100200700ff06000008ee09060100340700ff06000008e609060100480700ff06000008dd09060000010000ff090c07e709040111132dffffc40009060100010800ff0600bc0d9009060100020800ff060000000009060100030800ff06001c05dc09060100040800ff06000fed42acfd7e

As far as I can see there is more data in the header then on other meters.

Hello,

im trying to connect with the meter via rs converter. It should be 7E1 or 8N1 in config ?
Is encryption key and authentication key necessery to read from the meter ?

Can't get the installation to work. Tried pip, pip3, upgrading pip, manual installation of rust as well as the different suggestions after installation. Result shows
Python: 3.9.2
platform: Linux-5.15.61-v7l+-armv7l-with-glibc2.31
pip: n/a
setuptools: 65.6.3
setuptools_rust: 1.5.2
rustc: 1.66.0 (69f9c33d7 2022-12-12)

error: command '/usr/bin/arm-linux-gnueabihf-gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for cryptography
Failed to build cryptography
ERROR: Could not build wheels for cryptography, which is required to install pyproject.toml-based projects

Any further suggestions?

Some meter equipment are pushing data using unnumbered information frames on the local customer interface.
To be able to parse the data we should add support for parsing the frame. There is no need to add it to the HDLC state since it should not happen during normal communication with the meter over HDLC.

https://en.wikipedia.org/wiki/High-Level_Data_Link_Control

Hello,

I'm trying to read the data from my ISKRA AM550 trough the P1 port. The setup at the time is with a FT232 Serial cable into a raspberry py. The meter is provided by SIE Lausanne, in Switzerland.

The tiny doc they provided me is quite clear (though in french) ISKRA.pdf

Interface: P1 DSMR v5.0
Protocole: DLMS/COSEM
Frame length: 154 bytes

Baudrate: 1152000, 8bits, no parity, 1 stop bit.

Then they give some OBIS codes with what they match (e.g. current on phase 2, etc.)

They also state that the data are not encrypted.

I've tried all kind of libraries to read the data but with no success. I wrote a minimal example with your library:

from dlms_cosem.client import DlmsClient
from dlms_cosem import cosem, enumerations

from dlms_cosem.io import SerialIO, HdlcTransport

from dlms_cosem.security import NoSecurityAuthentication

usb_port: str = "/dev/ttyUSB0"


serial_io = SerialIO(
    port_name=usb_port,
    baud_rate=115200,
)
public_hdlc_transport = HdlcTransport(
    client_logical_address=1,
    server_logical_address=16,
    server_physical_address=17,
    io=serial_io,
)

public_client = DlmsClient(
    transport=public_hdlc_transport,
    authentication=NoSecurityAuthentication(),
)

with public_client.session() as client:
    pass

But the connection cannot starts and it look quite like one of the other issue that has been open: #62. I know the problem might not be on your side but I hope you can put me on some track. Many thanks.

2024-01-07 16:15:49 [debug    ] HDLC state transitioned        new_state=AWAITING_CONNECTION old_state=NOT_CONNECTED
2024-01-07 16:15:49 [debug    ] Sending data                   data=bytearray(b'~\xa0\x08 #\x03\x93\x1b\xc2~') transport=HdlcTransport(client_logical_address=1, server_logical_address=16, io=SerialIO(port_name='/dev/ttyUSB0', baud_rate=115200, timeout=10, serial_port=Serial<id=0x7f9cbaa2e0, open=True>(port='/dev/ttyUSB0', baudrate=115200, bytesize=8, parity='N', stopbits=1, timeout=10, xonxoff=False, rtscts=False, dsrdtr=False)), server_physical_address=17, client_physical_address=None, extended_addressing=False, timeout=10, hdlc_connection=HdlcConnection(client_address=HdlcAddress(logical_address=16, physical_address=17, address_type='server', extended_addressing=False), server_address=HdlcAddress(logical_address=1, physical_address=None, address_type='client', extended_addressing=False), client_ssn=0, client_rsn=0, server_ssn=0, server_rsn=0, max_data_size=128, state=HdlcConnectionState(current_state=AWAITING_CONNECTION), buffer=bytearray(b''), buffer_search_position=1), _send_buffer=[], out_buffer=bytearray(b''), in_buffer=bytearray(b''))

2024-01-07 16:15:54 [debug    ] Received data                  data=b'~\xa8\xa4\xcf\x02#\x03\x99\x96\xe6\xe7\x00\x0f\x00\x17\xc4@\x0c\x07\xe8\x01\x07\x07\x10\x0f7\x00\xff\xc4\x00\x02\x10\x02\x02\t\x06\x00\x00`\x01\x00\xff\t\x0885738459\x02\x02\t\x06\x00\x00`\x01\x01\xff\t\x071058708\x02\x03\t\x06\x01\x00\x01\x07\x00\xff\x06\x00\x00\x05\x9d\x02\x02\x0f\x00\x16\x1b\x02\x03\t\x06\x01\x00\x02\x07\x00\xff\x06\x00\x00\x00\x00\x02\x02\x0f\x00\x16\x1b\x02\x03\t\x06\x01\x00\x01\x08\x00\xff\x06\x00L\x9f\xe7\x02\x02\x0f\x00\x16\x1e\x02\x03\t\x06\x01\x00\x02\x08\x00\xff\x06\x00sk\xf7\x02\x02\x0f\x00\x16\x1e\x02\x03\t\x06\x01\x00 \x07\xbd\x0c~' transport=HdlcTransport(client_logical_address=1, server_logical_address=16, io=SerialIO(port_name='/dev/ttyUSB0', baud_rate=115200, timeout=10, serial_port=Serial<id=0x7f9cbaa2e0, open=True>(port='/dev/ttyUSB0', baudrate=115200, bytesize=8, parity='N', stopbits=1, timeout=10, xonxoff=False, rtscts=False, dsrdtr=False)), server_physical_address=17, client_physical_address=None, extended_addressing=False, timeout=10, hdlc_connection=HdlcConnection(client_address=HdlcAddress(logical_address=16, physical_address=17, address_type='server', extended_addressing=False), server_address=HdlcAddress(logical_address=1, physical_address=None, address_type='client', extended_addressing=False), client_ssn=0, client_rsn=0, server_ssn=0, server_rsn=0, max_data_size=128, state=HdlcConnectionState(current_state=AWAITING_CONNECTION), buffer=bytearray(b''), buffer_search_position=1), _send_buffer=[], out_buffer=bytearray(b''), in_buffer=bytearray(b''))
2024-01-07 16:15:54 [debug    ] Added data to buffer           data=b'~\xa8\xa4\xcf\x02#\x03\x99\x96\xe6\xe7\x00\x0f\x00\x17\xc4@\x0c\x07\xe8\x01\x07\x07\x10\x0f7\x00\xff\xc4\x00\x02\x10\x02\x02\t\x06\x00\x00`\x01\x00\xff\t\x0885738459\x02\x02\t\x06\x00\x00`\x01\x01\xff\t\x071058708\x02\x03\t\x06\x01\x00\x01\x07\x00\xff\x06\x00\x00\x05\x9d\x02\x02\x0f\x00\x16\x1b\x02\x03\t\x06\x01\x00\x02\x07\x00\xff\x06\x00\x00\x00\x00\x02\x02\x0f\x00\x16\x1b\x02\x03\t\x06\x01\x00\x01\x08\x00\xff\x06\x00L\x9f\xe7\x02\x02\x0f\x00\x16\x1e\x02\x03\t\x06\x01\x00\x02\x08\x00\xff\x06\x00sk\xf7\x02\x02\x0f\x00\x16\x1e\x02\x03\t\x06\x01\x00 \x07\xbd\x0c~'
2024-01-07 16:15:54 [debug    ] HDLC frame could not be parsed. Need more data
2024-01-07 16:15:54 [debug    ] Received data                  data=b'~\xa8\xa4\xcf\x02#\x03\x99\x96\x00\xff\x12\x08\xd8\x02\x02\x0f\xff\x16#\x02\x03\t\x06\x01\x004\x07\x00\xff\x12\x08\xe1\x02\x02\x0f\xff\x16#\x02\x03\t\x06\x01\x00H\x07\x00\xff\x12\x08\xc5\x02\x02\x0f\xff\x16#\x02\x03\t\x06\x01\x00\x1f\x07\x00\xff\x12\x00U\x02\x02\x0f\xfe\x16!\x02\x03\t\x06\x01\x003\x07\x00\xff\x12\x006\x02\x02\x0f\xfe\x16!\x02\x03\t\x06\x01\x00G\x07\x00\xff\x12\x02\xce\x02\x02\x0f\xfe\x16!\x02\x03\t\x06\x01\x00\x01\x08\x01\xff\x06\x00"\x90h\x02\x02\x0f\x00\x16\x1e\x02\x03\t\x06\x01\x00\x01\x08\x02\xff\x06\x00*\x0f\x7f\x02\x02\x0f\x00\x16\x1e\x02\x03\t\x06\x01\x00g\xf9~' transport=HdlcTransport(client_logical_address=1, server_logical_address=16, io=SerialIO(port_name='/dev/ttyUSB0', baud_rate=115200, timeout=10, serial_port=Serial<id=0x7f9cbaa2e0, open=True>(port='/dev/ttyUSB0', baudrate=115200, bytesize=8, parity='N', stopbits=1, timeout=10, xonxoff=False, rtscts=False, dsrdtr=False)), server_physical_address=17, client_physical_address=None, extended_addressing=False, timeout=10, hdlc_connection=HdlcConnection(client_address=HdlcAddress(logical_address=16, physical_address=17, address_type='server', extended_addressing=False), server_address=HdlcAddress(logical_address=1, physical_address=None, address_type='client', extended_addressing=False), client_ssn=0, client_rsn=0, server_ssn=0, server_rsn=0, max_data_size=128, state=HdlcConnectionState(current_state=AWAITING_CONNECTION), buffer=bytearray(b'~\xa8\xa4\xcf\x02#\x03\x99\x96\xe6\xe7\x00\x0f\x00\x17\xc4@\x0c\x07\xe8\x01\x07\x07\x10\x0f7\x00\xff\xc4\x00\x02\x10\x02\x02\t\x06\x00\x00`\x01\x00\xff\t\x0885738459\x02\x02\t\x06\x00\x00`\x01\x01\xff\t\x071058708\x02\x03\t\x06\x01\x00\x01\x07\x00\xff\x06\x00\x00\x05\x9d\x02\x02\x0f\x00\x16\x1b\x02\x03\t\x06\x01\x00\x02\x07\x00\xff\x06\x00\x00\x00\x00\x02\x02\x0f\x00\x16\x1b\x02\x03\t\x06\x01\x00\x01\x08\x00\xff\x06\x00L\x9f\xe7\x02\x02\x0f\x00\x16\x1e\x02\x03\t\x06\x01\x00\x02\x08\x00\xff\x06\x00sk\xf7\x02\x02\x0f\x00\x16\x1e\x02\x03\t\x06\x01\x00 \x07\xbd\x0c~'), buffer_search_position=166), _send_buffer=[], out_buffer=bytearray(b''), in_buffer=bytearray(b''))
2024-01-07 16:15:54 [debug    ] Added data to buffer           data=b'~\xa8\xa4\xcf\x02#\x03\x99\x96\x00\xff\x12\x08\xd8\x02\x02\x0f\xff\x16#\x02\x03\t\x06\x01\x004\x07\x00\xff\x12\x08\xe1\x02\x02\x0f\xff\x16#\x02\x03\t\x06\x01\x00H\x07\x00\xff\x12\x08\xc5\x02\x02\x0f\xff\x16#\x02\x03\t\x06\x01\x00\x1f\x07\x00\xff\x12\x00U\x02\x02\x0f\xfe\x16!\x02\x03\t\x06\x01\x003\x07\x00\xff\x12\x006\x02\x02\x0f\xfe\x16!\x02\x03\t\x06\x01\x00G\x07\x00\xff\x12\x02\xce\x02\x02\x0f\xfe\x16!\x02\x03\t\x06\x01\x00\x01\x08\x01\xff\x06\x00"\x90h\x02\x02\x0f\x00\x16\x1e\x02\x03\t\x06\x01\x00\x01\x08\x02\xff\x06\x00*\x0f\x7f\x02\x02\x0f\x00\x16\x1e\x02\x03\t\x06\x01\x00g\xf9~'
2024-01-07 16:15:54 [debug    ] HDLC frame could not be parsed. Need more data

And it goes on like this forever with more and more data in the buffer.

(base) anon$:~/anaconda3/lib/python3.10/site-packages/dlms_cosem$ python
Python 3.10.9 (main, Mar 1 2023, 18:23:06) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

from dlms_cosem import cosem, enumerations, utils, security
from dlms_cosem.client import DlmsClient
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'dlms_cosem.client'

On Profile generic the capture_period is stored in second. But in ProfileGenericBufferParser the capture_period is in minutes.

ProfileGenericBufferParser should be changed to seconds so that the value from a meter can be used directly without transformation into minutes.

To be able to set time in a meter we need to implement the SET service. Both the APDU parsing and the Client implementation.

We assume that we can do HLS auth agains assiciation object 0.0.40.0.0.255.

But other meters have other objects depending on the client used. (for example Installer client on italian meters)

So this should be a settings on the client. with previous value as default.

On typing extensions is not included in required packages.

It is possible for device to not use ReleaseRequest and ReleaseResonse.

Before this was handled by customizing the client, but since we now Have the DlmsConnectionSetting-object it would be better to put it there.

Example:

@attr.s(auto_attribs=True)
class DlmsConnectionSettings:
    """
    Class to hold values and settings to handle different quirks of different dlms
    server implementations and manufacturers specific irregularity.
    """
    # Indicates if the connection and the client should allow for RLRQ to be send.
    use_rlrq_rlre: bool = attr.ib(default=True)
    # In Pietro Fiorentini local communication over HDLC the system title in GeneralGlobalCiphering is omitted.
    empty_system_title_in_general_glo_ciphering: bool = attr.ib(default=False)

Suppport TCP/IP connections

I would like to see a simple example that just reads hdlc from serial port. I see there is already an example for parsing norwegian han. I would like to actually read from such a meter. It does not seem to need any setup, it sends data without any negotiations.

Add the GET.WITH_LIST service.

Add APDUS, update APDU-facories.
Add STATE.

API in client should be a separate method from get

Example:

def get_many(attributes: List[cosem.CosemAttributes]) -> List[DlmsData]:
   ...

When using the GET.WITH_LIST service the reponse is not parsed properly if the response contains an attribut that holds and object. (Data structure or Array)

HAN_aidon_rj45.zip

Hello,
I have made a small example for how to read data from Skellefteå Krafts meters with the RJ45 port.
Python3, an old Raspberry and a m-bus client card from Mikroe.

Hope it can be useful for someone.

Local communications on Pietro Fiorentini RSE gas meters do not use the system title in general-glo-ciphering APDU.

It is questionable if this is correct behaviour but we need to be pragmatic and solve it.

The system title is till used for the ciphered text, otherwise the encryption is not done correctly.

GeneralGlobalCiphering APDU is created in DlmsConnection.protect() and it would be possible to omit the system title but there also needs to be changes in the APDU class to allow for system_title=None and to encode it correctly into the byte representation.

This should not be default behaviour and we should introduce an optional argument to the DlmsConnection that handles complext settings for the connection. This could also adress the problem in #70
The more meters we aim to support the the more special cases we will find and it is better to have a serialziable object so we can easily transfer the settings in different environments and use cases.

For example

DlsmConnectionSettings:
   empty_system_title_in_general_glo_ciphering_apdu: bool = field(default=False)

The current readme is not up to date on the features.

The state management only handles ACTION as a part of the HLS flow.
We need to add general handling in the state for ACTION-request and responses.

The following data for a GetResponseWithList was not possible to parse:
b'\xc4\x03\xc1\x02\x00\x00\x02\x05\x12\x02\xd0\x12\x0c\xa8\x11\x18\x11\x1f\x0f\xc0'

How should we solve selective access?

It is appended on GET-requests and might have different structure depending on the attribute of the interface class we are trying to get.

Hello,
It would be nice to add "Short name referencing" association method.
I have PAFAL 16EC3rn/1 meter and when I try to associate with it I get error:

"ValueError("vaa-name in InitateResponse is not \x00\x07")".

I comented out this line and now I have error:

"AARE requests use of Short Name referencing which is not " "supported.".

Hope you will keep developing this great project ! 😄

I have been trying to use this but every time i run "examples/dlms_with_tcp_example.py" , i get this error:
from dlms_cosem import WrapperHeader
ImportError: cannot import name 'WrapperHeader' from 'dlms_cosem'

I tried changing dlms to dlms_cosem, but the problem still persists. Is there any other version that supports WrapperHeader?

There seems to be no documentation stating that ExceptionsResponses should be encrypted when you have entered an encrypted/protected context via association negotiation.
It has been seen that meters send unencrypted ExceptionsResponses for a GET service if the inparameters are wrong.

Currently, it is implemented:

• general-glo-ciphering [219]
• glo-initiateResponse [40]
• glo-initiateRequest [33]

but the following APDUs are missing:

-- with global ciphering

      glo-confirmedServiceError            [46]     IMPLICIT    OCTET STRING,


-- with dedicated ciphering

      ded-initiateRequest                  [65]     IMPLICIT    OCTET STRING,
      ded-initiateResponse                 [72]     IMPLICIT    OCTET STRING,
      ded-confirmedServiceError            [78]     IMPLICIT    OCTET STRING,

-- with global ciphering

    glo-get-request                      [200]   IMPLICIT     OCTET   STRING,
    glo-set-request                      [201]   IMPLICIT     OCTET   STRING,
    glo-event-notification-request       [202]   IMPLICIT     OCTET   STRING,
    glo-action-request                   [203]   IMPLICIT     OCTET   STRING,

    glo-get-response                     [204] IMPLICIT       OCTET STRING,
    glo-set-response                     [205] IMPLICIT       OCTET STRING,
    glo-action-response                  [207] IMPLICIT       OCTET STRING,

-- with dedicated ciphering

    ded-get-request                      [208]   IMPLICIT     OCTET   STRING,
    ded-set-request                      [209]   IMPLICIT     OCTET   STRING,
    ded-event-notification-request       [210]   IMPLICIT     OCTET   STRING,
    ded-actionRequest                    [211]   IMPLICIT     OCTET   STRING,

    ded-get-response                     [212] IMPLICIT       OCTET STRING,
    ded-set-response                     [213] IMPLICIT       OCTET STRING,
    ded-action-response                  [215] IMPLICIT       OCTET STRING,

-- general APDUs
    general-ded-ciphering                [220]   IMPLICIT     General-Ded-Ciphering,
    general-ciphering                    [221]   IMPLICIT     General-Ciphering,
    general-signing                      [223]   IMPLICIT     General-Signing,

In one of my tests, I have a DLMS server (over TCP) which responds slowly. This results in the following error message:
Traceback (most recent call last):
File "", line 1, in
File "C:\Program Files (x86)\Python38-32\lib\site-packages\dlms_cosem\clients\dlms_client.py", line 185, in get
self.send(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\dlms_cosem\clients\dlms_client.py", line 327, in send
response_bytes = self.io_interface.send(data)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\dlms_cosem\clients\blocking_tcp_transport.py", line 92, in send
return self.recv()
File "C:\Program Files (x86)\Python38-32\lib\site-packages\dlms_cosem\clients\blocking_tcp_transport.py", line 101, in recv
header = WrapperHeader.from_bytes(self.tcp_socket.recv(8))
File "C:\Program Files (x86)\Python38-32\lib\site-packages\dlms_cosem\protocol\wrappers.py", line 62, in from_bytes
raise ValueError(
ValueError: Wrapper Header can only consists of 8 bytes and got 1

I found a way to fix this in blocking_tcp_transport.py:
def recv(self) -> bytes:
"""
Receives a whole DLMS APDU. Gets the total length from the DLMS IP Wrapper.
"""
if not self.tcp_socket:
raise RuntimeError("TCP transport not connected.")
try:
header = WrapperHeader.from_bytes(self.read_fixed_length(self.tcp_socket, 8))
data = self.read_fixed_length(self.tcp_socket, header.length)
except (OSError, IOError, socket.timeout, socket.error) as e:
raise exceptions.CommunicationError("Could not receive data") from e
return data

def read_fixed_length(self, sckt, length):
    data=b''
    while len(data) < length:
        data += sckt.recv(length-len(data))
    return data

The current A XDR parser was made so that it was possible to just write a parsing config to APDUs and it would output the correct dict to create the object.

But it has proven that it is just was smoother to handparse each APDU. Some of the larger and more complex ones might be a little bit annoying but on the whole it is much clearer in the code when the parsing is done explicitly.

The use we want to keep it for is to parse dynamic DLMS data returned from meters. If we remove the general parsing and make it into a dlms response data parse we can get a smoother solution than:

def parse_as_dlms_data(data: bytes):
    data_decoder = a_xdr.AXdrDecoder(
        encoding_conf=a_xdr.EncodingConf(
            attributes=[a_xdr.Sequence(attribute_name="data")]
        )
    )
    return data_decoder.decode(data)["data"]

As of now we can only parse an OBIS code from a dotted representation like 0.0.1.8.0.255

But the "normal" way of writing the obis would be 0-0:1.8.0.255 or even 0-0:1.8.0 as the last 255 usually can be omitted (but not in the byte representation)

What we would like is a from_string method that can handle any configuration.

0-0:1.8.0.255 can easily be parsed with regex.
0-0:1.8.0 should also be parsed with regex.

We might even be able to specify that we can have any delimiter. This could come in handy as we in some application have to use 0-0-1-8-0-255 as dot has special meaning in strings.

Some older meters using IEC62056-21 only uses a 3 char representation like 1.8.0 but this should no be allowed as we would not be able to transform back and forth with this.

We should also change the method to_string to have a standard format of the "normal" representation with a option to
add a custom delimiter:

def to_string(delimiter: Optional[str] = None)
    if delimiter is not None:
        return f"{a}-{b}:{c}.{e}.{f}.{g}"
    else:
        return  f"{a}{delimiter}{b}{delimiter}{c}{delimiter}{e}{delimiter}{f}{delimiter}{g}"

Hey, I have a strange issue where im getting some characters that maybe shouldn't be there in the output of a get request.

Here is a part of a script table I have read from the meter:

[[1, [[1, 6, bytearray(b'\x00\x00\x0e\x00\x01\xff'), 4, bytearray(b'\x01')], 
[1, 6, bytearray(b'\x00\x00\x0e\x00\x02\xff'), 4, bytearray(b'\x01')], 
[1, 70, bytearray(b'\x00\x01`\x03\n\xff'), 3, 2]]], 

And here is part of the IDIS table on how it should look like:

  {1,{
          {1,6,0-0:14.0.1.255,4,*},
          {1,6,0-0:14.0.2.255,4,*},
          {2,70,0-1:96.3.10.255,[1..2],0}
        }
   },

Now all the values here are correct except the last OBIS code which has an "`" and "n" in it and has missing parts.

As I understand it, European HAN energy meters (both according to Swedish, Norwegian, and Dutch standard) typically does not use encryption. Is it possible in any easy way to get rid of the cryptography dependency? This one in turn depends on quite a lot of other packages.

when reading attributes on certain interface classes the top element is a structure.

Right now the element count seems to be interpreted as a data tag.

data = b'\x02\x07\x06\x00\x00\x00\x00\x12\x00\x00\x11\x00\x11\x00\x12\x00\x00\x12\x00\x00\x12\x00\x00'

 return cls.MAP[tag]
KeyError: 7

from dlms_cosem.clients.serial_hdlc import SerialHdlcClient

Should serial_hdlc be hdlc_transport and SerialHdlcClienthere be SerialHdlcTransport, or is there a missing client?

The default timeout is 10 seconds, but it will wait forever

I did a test as requested on https://hanporten.se/norska/protokollet/ with a live Kamstrup meter and got "NotImplementedError: VisibleStringData have not implemented byte conversion".

The HDLC data is here:

7ea0e22b2113239ae6e7000f000000000c07e60c1c03170100ff80000002190a0e4b616d73747275705f563030303109060101000005ff0a103537303635363731393633383234383509060101600101ff0a1236383431313331424e31343331303130393009060101010700ff060000092609060101020700ff060000000009060101030700ff060000000009060101040700ff06000000df090601011f0700ff060000005c09060101330700ff060000008d09060101470700ff060000037209060101200700ff1200e609060101340700ff1200e609060101480700ff1200e49a187e

This is the included data:

{
  "obis_list_version": "Kamstrup_V0001",
  "meter_ID": "5706567196382485",
  "meter_model": "6841131BN143101090",
  "act_pow_pos": 2342,
  "act_pow_neg": 0,
  "react_pow_pos": 0,
  "react_pow_neg": 223,
  "curr_L1": 92,
  "curr_L2": 141,
  "curr_L3": 882,
  "volt_L1": 230,
  "volt_L2": 230,
  "volt_L3": 228
}

The XADRParser is now returning the value if an attribute is just a single value and a list of values if there is more values.
This made it annoying building a more general way reading alot of values.

It would be better if the function returned a list in either case. If it is a single value you will need to access the first element in the list instead.

Hello

Reset and capture methods of ProfileGeneric class are void. It it possible to implement them?
Regards

Hi Henrik,

Thanks for your DLMS package, it has been proved very useful.

Issue
I found a bug in dlms_cosem/dlms_data.py when converting the COSEM DataArray object to bytes. Atm, it appends to its bytearray from encode_variable_integer which returns bytes, this raises:

TypeError: 'bytes' object cannot be interpreted as an integer

I guess the intention is to extend as done in DataStructure.

Proposal

Change:

class DataArray(BaseDlmsData):
    """Sequence of Data"""

    TAG = 1
    LENGTH = VARIABLE_LENGTH

    def to_bytes(self) -> bytes:
        out = bytearray()
        out.append(self.TAG)
        out.append(encode_variable_integer(len(self.value)))

to:

class DataArray(BaseDlmsData):
    """Sequence of Data"""

    TAG = 1
    LENGTH = VARIABLE_LENGTH

    def to_bytes(self) -> bytes:
        out = bytearray()
        out.append(self.TAG)
        out.extend(encode_variable_integer(len(self.value)))

I don't see much movement in recent months. I like this project because you see mastery in Python and a depth knowledge of the standard. Plus, the implementation is enjoyable, not only for what it does, but also for how it does it

If it is maintained, I could do the effort of creating various merge requests for the following commits in my fork and further contribute. Some of those changes are:

• Set request with blocks (only parsing with tests)
• Association Release user_information fix

I am trying to access a Kamstrup devices over a IR eye.

According to Kamstrup it should with with DLMS and have a default password of 12345.

Connecting
16:40:28,576 : DEBUG : HDLC state transitioned from NOT_CONNECTED to AWAITING_CONNECTION
16:40:28,577 : DEBUG : Sending: bytearray(b"~\xa0\x08 \x03\'\x93s\x85~")
16:40:28,604 : DEBUG : Received HDLC data: b"~\xa0\x08 \x03'\x93s\x85~"
16:40:28,605 : DEBUG : HDLC frame could not be parsed. Need more data

Minimal sample

import logging
from functools import partial
from pprint import pprint

from dateutil import parser as dateparser

from dlms_cosem import enumerations
from dlms_cosem.clients.dlms_client import DlmsClient

# set up logging so you get a bit nicer printout of what is happening.
logging.basicConfig(
    level=logging.DEBUG,
    format="%(asctime)s,%(msecs)d : %(levelname)s : %(message)s",
    datefmt="%H:%M:%S",
)

auth = enumerations.AuthenticationMechanism.LLS


public_client = partial(
    DlmsClient.with_serial_hdlc_transport,
    server_logical_address=16,
    server_physical_address=1,
    client_logical_address=19,
)

management_client = partial(
    DlmsClient.with_serial_hdlc_transport,
    server_logical_address=16,
    server_physical_address=1,
    client_logical_address=19,
    authentication_method=auth,
    password=b"12345"
)

port = "/dev/ttyUSB1"


print ("Connecting")

with management_client(
    serial_port=port
).session() as client:
    print ("Connected")

    print(client.dlms_connection.meter_system_title.hex())
    # print(result[0][0].value.isoformat())

I am new to dlms so I am probably missing something terribly simple, I have aligned the logical addresses with the documentation. The electrical company has sent me the DLMS documentation

55121424_D1_GB.pdf DLMS generation M N1.pdf

Application context should only be available in the RLRQ and RLRE if the AA has been set up with ciphering.
So for example when using the public client it should not be present.

Furthermore
In the the RLRQ the original proposed dlms context should be used. And the meter/device/server should answer in the RLRE with the original negotiated dlms context.

As stated in Green Book 9.3.3.2

The API as of now is more focused on for one connection to only connect to a meter with on type of client.
For example the public or the management client.

But when reading an "unknown" meter we might not know the invocation counter to be used for encryption by the management client and it has to be read by the public client first.

As of now that would involve creating a connection to the meter with the managment client, getting and error and disconnecting. Then creating a new connection to the meter with the public client, getting the invocation counter and disconnecting. Then creating another connection to the meter with the management client and correct invocation counter.

This introduces delays and it can also lead to locking a modem of some meters.

There is no real need to disconnect a transport between switching context (client types). Since that is handled by the association and associon release in the DLMS communication.
So it is way more efficient to keep the socket open and reuse it for each client type.

Or actually we would like to be able to change the client into a new type instead of having several objects that share the transport object. This could lead to calls from different clients on the transport and might mess stuff up.

But when we change the clients logical address we need to update the transports too.

Hey, I am trying to build a testing tool for a meter from Mikroelektronika MEM600. Now I have the company tool which communicates with the meter over DLMS HDLC and I can read and write all the registers from there(obviously I have the auth information necessary to access the meter with the 01 management client). Now I want to build a quick tool to read and sometimes write registers of the meter but I cant even get it to start communicating.

I am using the your HDLC example for testing(examples/dlms_with_hdlc_example.py) only modified the serial port to COM9 since that is where my optical probe is, and it is sending out the first message but the meter is not responding ( I know this meter works and communication works because it works with our software and I can verify that communication thru the optical and RS485 ports is set to DLMS(not mod E)).

Here is the output I get:

2024-01-22 13:00:11 [debug    ] HDLC state transitioned        new_state=AWAITING_CONNECTION old_state=NOT_CONNECTED
2024-01-22 13:00:11 [debug    ] Sending data                   data=bytearray(b'~\xa0\x08\x02#!\x93\xbdd~') transport=HdlcTransport(client_logical_address=16, server_logical_address=1, io=SerialIO(port_name='COM9', baud_rate=9600, timeout=10, serial_port=Serial<id=0x20eb7ea4070, open=True>(port='COM9', baudrate=9600, bytesize=8, parity='N', stopbits=1, timeout=10, xonxoff=False, rtscts=False, dsrdtr=False)), server_physical_address=17, client_physical_address=None, extended_addressing=False, timeout=10, hdlc_connection=HdlcConnection(client_address=HdlcAddress(logical_address=1, physical_address=17, address_type='server', extended_addressing=False), server_address=HdlcAddress(logical_address=16, physical_address=None, address_type='client', extended_addressing=False), client_ssn=0, client_rsn=0, server_ssn=0, server_rsn=0, max_data_size=128, state=HdlcConnectionState(current_state=AWAITING_CONNECTION), buffer=bytearray(b''), buffer_search_position=1), _send_buffer=[], out_buffer=bytearray(b''), in_buffer=bytearray(b''))
2024-01-22 13:00:21 [debug    ] Received data                  data=b'' transport=HdlcTransport(client_logical_address=16, server_logical_address=1, io=SerialIO(port_name='COM9', baud_rate=9600, timeout=10, serial_port=Serial<id=0x20eb7ea4070, open=True>(port='COM9', baudrate=9600, bytesize=8, parity='N', stopbits=1, timeout=10, xonxoff=False, rtscts=False, dsrdtr=False)), server_physical_address=17, client_physical_address=None, extended_addressing=False, timeout=10, hdlc_connection=HdlcConnection(client_address=HdlcAddress(logical_address=1, physical_address=17, address_type='server', extended_addressing=False), server_address=HdlcAddress(logical_address=16, physical_address=None, address_type='client', extended_addressing=False), client_ssn=0, client_rsn=0, server_ssn=0, server_rsn=0, max_data_size=128, state=HdlcConnectionState(current_state=AWAITING_CONNECTION), buffer=bytearray(b''), buffer_search_position=1), _send_buffer=[], out_buffer=bytearray(b''), in_buffer=bytearray(b''))

It just constantly keeps receiving "empty" messages.

I am not too familiar with HDLC and DLMS so I might be missing something. I have tried different addresses, using the 16(client) address, I have tried different physical addresses.

And here is the communication that occurs between our software and the meter:

Open : COM9 (22/01/2024 13:18:40 )
TX - 
7E A0 0A 00 02 3A 71 03 53 90 01 7E ( +  41004 ms)  
RX - 
7E A0 0A 03 00 02 3A 71 1F 83 D6 7E ( +  437 ms)  
TX - 
7E A0 0A 00 02 3A 71 03 93 9C C7 7E ( +  226 ms)  
RX - 
7E A0 23 03 00 02 3A 71 73 53 5D 81 80 14 05 02 00 F2 06 02 00 F2 07 04 00 00 00 01 08 04 00 00 00 01 0C F7 7E ( +  196 ms)  
TX - 
7E A0 47 00 02 3A 71 03 10 46 98 E6 E6 00 60 36 A1 09 06 07 60 85 74 05 08 01 01 8A 02 07 80 8B 07 60 85 74 05 08 02 01 AC 0A 80 08 31 32 33 34 35 36 37 38 BE 10 04 0E 01 00 00 00 06 5F 1F 04 00 FF FF FF FF FF AD 1A 7E ( +  216 ms)  
RX - 
7E A0 3A 03 00 02 3A 71 30 3C D9 E6 E7 00 61 29 A1 09 06 07 60 85 74 05 08 01 01 A2 03 02 01 00 A3 05 A1 03 02 01 00 BE 10 04 0E 08 00 06 5F 1F 04 00 00 FA 1D 00 EF 00 07 D5 05 7E ( +  458 ms)  
TX - 
7E A0 1C 00 02 3A 71 03 32 1B 9E E6 E6 00 C0 01 C1 00 01 00 00 60 01 00 FF 02 00 89 A0 7E ( +  281 ms)  
RX - 
7E A0 1D 03 00 02 3A 71 52 B3 3D E6 E7 00 C4 01 C1 00 09 08 31 31 32 32 33 37 36 38 C5 46 7E ( +  232 ms)  
TX - 
7E A0 0A 00 02 3A 71 03 51 82 22 7E ( +  237 ms)  
RX - 
7E A0 0A 03 00 02 3A 71 51 F9 7D 7E ( +  170 ms)  

when i ran the code, i got this error "raise DataResultError(
dlms_cosem.client.DataResultError: Could not perform GET request: <DataAccessResult.OBJECT_UNAVAILABLE: 11>".
I tried everything but just couldn't resolve it.