aclindsa / ofxgo Goto Github PK

Is there a list maintained somewhere for all the finance institutions that are tested with this library?

Though it won't be complete, the README should at least provide simple example code

Same problem for an AMEX and a Citi credit card account.

Example command:

ofx transactions-cc -username <redacted> \
    -url "https://online.americanexpress.com/myca/ofxdl/desktop/desktopDownload.do?request_type=nl_ofxdownload" \
    -org AMEX -fid 3101 -appid QWIN -appver 2600 -acctid <redacted>

Output:

Transactions:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6619fd]

goroutine 1 [running]:
main.ccTransactions()
	/home/martin/go/pkg/mod/github.com/aclindsa/[email protected]/cmd/ofx/cctransactions.go:63 +0x75d
main.runCmd(0xc000173f38)
	/home/martin/go/pkg/mod/github.com/aclindsa/[email protected]/cmd/ofx/main.go:57 +0x114
main.main()
	/home/martin/go/pkg/mod/github.com/aclindsa/[email protected]/cmd/ofx/main.go:69 +0x1e5

Thanks for releasing this library.

I'm excited to use it, but it's taking a long time to download. go get is getting stuck on downloading aclindsa/go, which seems to be a big repo.

Is it possible to use the default Go encoding/xml or put a fork in a smaller git repo?

query.URL = "https://www.oasis.cfree.com/fip/genesis/prod/02601.ofx"
query.Signon.Org = ofxgo.String("ISC")
query.Signon.Fid = ofxgo.String("2601")

I keep getting no bank messages received using the example. with my credentials and bank id and acctid in place.

The output of go get -v github.com/aclindsa/ofxgo/cmd/ofx is:

# github.com/aclindsa/xml
../../../github.com/aclindsa/xml/typeinfo.go:47: undefined: sync.Map```

My go version is `go version go1.8 darwin/amd64`.  Am I working in the right version for this to work?  Thanks!

Parsing OFX responses with SECURITY setting of TYPE1 fails with a validation error:

can't parse response: OFX SECURITY header not NONE

My bank still uses OFX 1.02 OFXSGML format, but uses SECURITY:TYPE1:

OFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:TYPE1
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE

This is parsed beautifully after I change that security setting to NONE:

OFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE

Apparently, this is a common issue in financial software processing .ofx responses, with the "fix" for users to hand-edit the responses to remove the TYPE1 security value and replace it with NONE:

• e.g. Statement Import Error (OFX Security Unsupported)

When I went digging to learn more about this setting I found Open Financial Exchange Specification 2.2, Nov 26, 2017 which states in section 4.2.2.2 Type 1 Protocol Overview that:

    Type 1 applies only to the request part of a message; the server response is unaffected.

Thus I think we could remove [this validation](-

Bank of Montreal (BMO) allows downloads of multiple accounts at once. It uses BANKMSGSETV1 element for that (with embedded BANKMSGSET inside it for some reason), but it emits start elements instead of end elements at the end 🤦‍♂️ .

OFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE

<OFX>
<SIGNONMSGSRSV1>
<SONRS>
<STATUS>
<CODE>0
<SEVERITY>INFO
<MESSAGE>OK
</STATUS>
<DTSERVER>20181228000224.466[-5:EDT]
<USERKEY>329402394029374
<LANGUAGE>ENG
<INTU.BID>00001
</SONRS>
</SIGNONMSGSRSV1>
<BANKMSGSETV1><BANKMSGSET>
<BANKMSGSRSV1>

<STMTTRNRS>
...
</STMTTRNRS>

</BANKMSGSRSV1>
<BANKMSGSET><BANKMSGSETV1>
</OFX>

It seems that the library currently doesn't support MSGSET elements at all, but even if it did the end elements would still be a problem. I could get the file through by making response parsing ignore MSGSET elements completely, as follows:

@@ -356,7 +357,13 @@ func ParseResponse(reader io.Reader) (*Response, error) {
                        return nil, err
                } else if ofxEnd, ok := tok.(xml.EndElement); ok && ofxEnd.Name.Local == "OFX" {
                        return &or, nil // found closing XML element, so we're done
+               } else if ofxEnd, ok := tok.(xml.EndElement); ok && strings.Contains(ofxEnd.Name.Local, "MSGSET") {
+                       continue // ignore MSGSET elements
                } else if start, ok := tok.(xml.StartElement); ok {
+                       // ignore MSGSET elements
+                       if strings.Contains(start.Name.Local, "MSGSET") {
+                               continue
+                       }
                        slice, ok := messageSlices[start.Name.Local]
                        if !ok {
                                return nil, errors.New("Invalid message set: " + start.Name.Local)

Note that I had to ignore EndElements as well, to allow the parser to unwind its element stack properly. The way the sample above is parsed is as if there are two sets of MSGSET elements nested in each other and the end elements get injected from the parser stack when it encounters the closing OFX element.

Anyway, I admit that this is outright disgusting, but it does seem to allow processing the file correctly including multiple account message sets. I absolutely understand if this is too much of a hack to allow in. I can muscle my way through outside of the ofxgo library if needed (just use some grep -v MSGSET equivalent), but I figured I'd bring it up anyway.

The OFX 102 spec is a bit lacking in this area, but it seems that some institutions require ofx elements not include their end tags (aggregates are ok though).

The key rule of Open Financial Exchange syntax is that each tag is either an element or an aggregate. Data follows its element tag. An aggregate tag begins a compound tag sequence, which must end with a matching tag; for example, <AGGREGATE> ... </AGGREGATE>.

From the reading above and my use of the python ofxclient, I think always omitting the element end tags should work for all institutions using OFX 102 (maybe all 100-series, but I haven't looked yet).

In my case, USAA doesn't like me using any element end tags. This is my hacky work-around:

var r *ofxgo.Request
b, _ := r.Marshal()

data := b.Bytes()
for _, s := range []string{"DTCLIENT", "DTSTART", "DTEND", "INCLUDE", "USERID", "USERPASS", "LANGUAGE", "ORG", "FID", "APPID", "APPVER", "TRNUID", "BANKID", "ACCTID", "ACCTTYPE"} {
	data = bytes.Replace(data, []byte("</"+s+">"), []byte{}, -1)
}
b = bytes.NewBuffer(data)

What do you think would be a good solution to this problem? I briefly looked at your fork of go's xml package, looks like that might be a good place for it. Maybe it could be some kind of encoder option?

Response from Fidelity:

<?xml version="1.0" encoding="UTF-8" standalone="no"?><?OFX OFXHEADER="200" VERSION="203" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="NONE"?><OFX><SIGNONMSGSRSV1><SONRS><STATUS><CODE>0</CODE><SEVERITY>INFO</SEVERITY><MESSAGE>SUCCESS</MESSAGE></STATUS><DTSERVER>20230906145854.013[-4:EDT]</DTSERVER><LANGUAGE>ENG</LANGUAGE><FI><ORG>fidelity.com</ORG><FID>7776</FID></FI></SONRS></SIGNONMSGSRSV1><BANKMSGSRSV1><STMTTRNRS><TRNUID>553f36a9-b492-8e8f-b86f-6284b76597f9<STATUS><CODE>0<SEVERITY>INFO<MESSAGE>SUCCESS</STATUS>...

At the tail end of that ... <MESSAGE>is indeed implicitly closed by </STATUS>. Is this a parsing problem? Or is Fidelity's response incorrectly formatted?

I'm currently using my personal Chase Bank checking, savings and credit accounts to play with this package. Most things are working as expected however I am not able to get my statement balances and transactions to be as up to date as I would like. It appears the responses ignore pending charges entirely regardless of whether the IncludePending flag is set to true. Have you seen this behavior before? I have to imagine retrieving pending charges would be supported by chase but I could be wrong.

It looks like some banks, like USAA, don't support standard unix line endings \n:

OFXAdapter: Failed to parse request: Unable to parse an atomic field "OFXRequest.Headers.OFXHeader": unable to find end delimiters "
".

Looking around, I found this issue which solved the problem by adding \r: euforic/banking.js#8

Based on the code in the python ofxclient library, it looks like using \r\n works for all banks: https://github.com/captin411/ofxclient/blob/v2.0.4/ofxclient/client.py#L25

In my custom client, I've hacked around the problem with this line:

var r *ofxgo.Request
b, _ := r.Marshal()
b = bytes.NewBuffer(bytes.Replace(b.Bytes(), []byte{'\n'}, []byte{'\r', '\n'}, -1))

What are your thoughts on changing the line ending for all requests? If you like I can try my hand at a PR too 😄