stellar-deprecated / horizon Goto Github PK

"I just took a look at the api and don't see any way to determine what is running on horizon. I think a good thing to add would be an api feature to determine what version of horizon is running and what steller-core version it is controling"

ERRO[0522] panic: sql: Scan error on column index 2: unsupported driver -> Scan pair: -> _string stacktrace=goroutine 43 [running]:
github.com/stellar/go-horizon.func·010()
/Users/andrewrogers/Code/go/src/github.com/stellar/go-horizon/middleware_recover.go:23 +0xd5
runtime.panic(0x56e4e0, 0xc2083fabe0)
/usr/local/go/src/pkg/runtime/panic.c:248 +0x18d
github.com/stellar/go-horizon/render.Collection(0xc1b230, 0xc2083b3ea0, 0xc1b3b8, 0xc2083b4c40, 0xc208312d00, 0xc1b490, 0xc2083b4c80, 0x7aa100)
/Users/andrewrogers/Code/go/src/github.com/stellar/go-horizon/render/main.go:41 +0x17c
github.com/stellar/go-horizon.offerIndexAction(0xc2083ee960, 0xc2083ee870, 0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/stellar/go-horizon/actions_offer.go:27 +0x390
github.com/zenazn/goji/web.handlerFuncWrap.ServeHTTPC(0x7aa0f8, 0xc2083ee960, 0xc2083ee870, 0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/zenazn/goji/web/handler.go:25 +0x54
github.com/zenazn/goji/web.(_router).route(0xc20802ce38, 0xc208288cc0, 0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/zenazn/goji/web/router.go:119 +0x143
github.com/zenazn/goji/web.func·002(0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/zenazn/goji/web/middleware.go:88 +0x5f
net/http.HandlerFunc.ServeHTTP(0xc20831b060, 0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/PuerkitoBio/throttled.func·003(0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/PuerkitoBio/throttled/throttler.go:64 +0x159
net/http.HandlerFunc.ServeHTTP(0xc20831b080, 0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/rs/cors.func·001(0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/rs/cors/cors.go:160 +0x185
net/http.HandlerFunc.ServeHTTP(0xc20831b0a0, 0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/zenazn/goji/web/middleware.func·003(0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/zenazn/goji/web/middleware/options.go:70 +0x12b
net/http.HandlerFunc.ServeHTTP(0xc20831b0c0, 0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/stellar/go-horizon.func·011(0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/stellar/go-horizon/middleware_recover.go:34 +0xf8
net/http.HandlerFunc.ServeHTTP(0xc20831b0e0, 0xc1b3b8, 0xc2083b4c40, 0xc208312d00)
/usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/stellar/go-horizon.func·012()
/Users/andrewrogers/Code/go/src/github.com/stellar/go-horizon/middleware_request_metrics.go:19 +0x86
github.com/rcrowley/go-metrics.(*StandardTimer).Time(0xc208081ad0, 0xc2083b3f20)
/Users/andrewrogers/Code/go/src/github.com/rcrowley/go-metrics/timer.go:212 +0x45
github.com/stellar/go-horizon.func·013(0xc1b3b8, 0xc2083b4b80, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/stellar/go-horizon/middleware_request_metrics.go:20 +0x1d1
net/http.HandlerFunc.ServeHTTP(0xc20831b100, 0xc1b3b8, 0xc2083b4b80, 0xc208312d00)
/usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/stellar/go-horizon.func·009(0xc1b150, 0xc208399a40, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/stellar/go-horizon/middleware_logger.go:26 +0x147
net/http.HandlerFunc.ServeHTTP(0xc20831b120, 0xc1b150, 0xc208399a40, 0xc208312d00)
/usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/sebest/xff.func·002(0xc1b150, 0xc208399a40, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/sebest/xff/xff.go:64 +0x86
net/http.HandlerFunc.ServeHTTP(0xc208318f40, 0xc1b150, 0xc208399a40, 0xc208312d00)
/usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/stellar/go-horizon.func·007(0xc1b150, 0xc208399a40, 0xc208312d00)
/Users/andrewrogers/Code/go/src/github.com/stellar/go-horizon/middleware_context.go:24 +0x2e8
net/http.HandlerFunc.ServeHTTP(0xc20831b140, 0xc1b150, 0xc208399a40, 0xc208312d00)
/usr/local/go/src/pkg/net/http/server.go:1235 +0x4

For ledgers that does not exist (yet) resources like /operations or /transactions return empty result set. For example for: https://horizon-testnet.stellar.org/ledgers/19384724/operations horizon's response is:

{
  "_embedded": {
    "records": []
  },
  "_links": {
    "next": {
      "href": "/ledgers/19384724/operations?order=asc\u0026limit=10\u0026cursor="
    },
    "prev": {
      "href": "/ledgers/19384724/operations?order=desc\u0026limit=10\u0026cursor="
    },
    "self": {
      "href": "/ledgers/19384724/operations?order=asc\u0026limit=10\u0026cursor="
    }
  }
}

It should respond with not_found problem.

Please update the documentation stating that transaction must be base64 encoded and not in hex.

Here is the error message while trying to get the master account info:

INFO[0010] Starting request                              method=GET path=/accounts/GCEZWKCA5VLDNRLN3RPRJMRZOX3Z6G5CHCGSNFHEYVXM3XOJMDS674JZ
DEBU[0010] Negotiated content type                       accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 content_type=application/hal+json
INFO[0010] Executing query                               sql=SELECT ha.* FROM history_accounts ha WHERE address = $1 LIMIT 1
INFO[0010] Executing query                               sql=SELECT a.accountid, a.balance, a.seqnum, a.numsubentries, a.inflationdest, a.thresholds, a.flags FROM accounts a WHERE accountid = $1 LIMIT 1
INFO[0010] Executing query                               sql=SELECT tl.accountid, tl.issuer, tl.alphanumcurrency, tl.tlimit, tl.balance, tl.flags FROM trustlines tl WHERE accountid = $1
ERRO[0010] pq: column tl.alphanumcurrency does not exist  file=src/github.com/stellar/go-horizon/render/problem/main.go line=65

Presently, we have to do a bunch of type assertions when rendering SSE events:

    if e, ok := e.(HasId); ok {
        fmt.Fprintf(w, "id: %s\n", e.SseId())
    }

    if e, ok := e.(HasEvent); ok {
        fmt.Fprintf(w, "event: %s\n", e.SseEvent())
    }

etc.

Instead, we should merge Event, HasId, HasEvent into the following rough code:

type Event struct {
  Data interface{}
  Id string
  Event string
  Retry int
}

type Eventer interface {
  Event() Event
}

This will reduce the number of methods an conforming type must implement and simplify the processing pipeline.

It always returns an empty collection

squirrel + sqlx seems to be a more flexible pair than GORM.

Calling it "main" conflicts with my other "mains"

Presently, every query for which a stream is requests gets registered on the streaming manager and is only then processed at each ledger close. This is wasteful, and running the query once during the initial request will:

1. Keep long-lived connections lower because queries that can be completely filled without new ledger data will complete within the http request
2. Provide faster quicker because we won't be waiting for a ledger close to perform the initial query.
3. Be more resilient because some results will be served to clients even if the stellar-core this horizon instance is cooperating with is stalled.

Getting the orderbook via the REST API /orderbooks is stil not working, a 404 is returned.

Hi @nullstyle I obviously lack a lot of knowledge about the goals of the project and the deliberations behind the choices made so far, and I'm also sure that you've considered a lot of Go libraries and frameworks already, so these are just my five cents and I'm assuming that I could be completely wrong on those 😄

I recently bumped into the Revel web app framework, and coming from a Django background (and I know you're a Rails guy), I felt this could be pretty useful to get some "standard web app type of stuff", and also to quickly establish patterns within Horizon that other devs could quickly grasp and build on.

Also I couldn't help but notice that you're already following some Rails patterns, and perhaps you'll feel that Revel could save you a lot of work in further implementing those.

Besides that, the GORM library, which is, well, an ORM, caught my eyes. Although on the DB side it seems there is much more "non-standard" stuff going on, so perhaps max flexibility is more important.

Anyway, just a bunch of ideas I wanted to throw around...

The unit tests still uses the base58 encoding, they should be ported to base32.

In the context of #76, #80, #86 I'd like to start a discussion about horizon DB denormalization.

horizon is a critical component in Stellar ecosystem. Because of this it should be very efficient, fast and, I hope, will have to server requests of millions of users. I'd like to start with a very simple solution that will minimize or even prevent possible performance issues but also make implementing features like those above much easier.

Let's denormalize all DB tables in a way that no 1:1 and 1:n relation JOINs are needed.

I understand that it will cost storage but storage is cheaper (exactly $0.100 per GB-month on RDS) than DB instances' CPU or Ram required to run expensive SQL queries. It will also save us a lot of time because we won't have to deal with performance issues that may arise with JOINs: we won't have to optimize queries and I think that for long we won't need a caching layer. horizon DB is almost read-only (I think only orderbook will change over time?) so it would be very easy to implement - since data won't change over time we won't run into issues like updating a lot of rows to reflect changes made in one table (check Example 3 below).

This idea is just an entry point to the discussion so add your comments.

Example 1 - no changes over time
In #80 we wanted to add information about account connected to effect. Existing database schema and a single row:

 history_account_id | history_operation_id | order | type |             details              
--------------------+----------------------+-------+------+----------------------------------
    105505871630336 |      105505871630337 |     1 |    0 | {"starting_balance": 1000000000}

The desired schema:

 history_account_id |                      account_id                          | history_operation_id | order | type |             details              
--------------------+----------------------------------------------------------+----------------------+-------+------+----------------------------------
    105505871630336 | GBS43BF24ENNS3KPACUZVKK2VYPOZVBQO2CISGZ777RYGOPYC2FT6S3K |     105505871630337  |     1 |    0 | {"starting_balance": 1000000000}

No JOINs are needed and account ID connected to this effect will never change - we're safe.

Example 2 - data changes over time
Account can have signers and we are going to display current signers in /accounts endpoint. We could create a account_signers table but more effective solution would be to simply add signers field to history_accounts table.
In this case horizon-importer will change signers field.

Example 3 - when it won't work
I can't find any example right now but to visualize it let's say users can change their account address and we won't display address of the account that sent a transaction. In this case, we obviously don't won't to denormalize history_transactions database because we would need to change a lot of records when user changes an address (1:n relation). In this case we should use JOIN.

Drawbacks

• Schema changes in large databases take a lot of time. In production, we will probably need to start a new DB and horizon-importer and switch to a new DB after import is done.
• ??

I'm not happy with how couple the render package is to the db package. render's methods presently have signatures that contain db.Query and Transform.

Instead, I propose we introduce a new concept, the ResourceProvider interface:

type ResourceProvider interface {
  GetResource(dest *interface{}) error  // populate dest with the first resource from the provider
  GetResources(dest []interface{}) error  // populate dest with the all resources from the provider
  StreamResources(dest *interface{}) error // populate dest with chan that streams resources
}

Prior to our production launch, we should go through the codebase and make sure we're using the problem renderer for all of our error responses to the client

A transaction to send lumen from one account to another is sent to go-horizon and it replies with the following message:

{ hash: '22fae4ad6ad65b7a99ee6f5167126d66b217a00afedf1068ec2fb15e7ab22d8b',
result: 'failed',
error: '0000000000000000fffffffb00000000' }

The source code for sending the transaction can be found here

Would it be possible to get a more meaningful error message ?

https://github.com/stellar/go-horizon/blob/master/src/github.com/stellar/go-horizon/resource_operation.go#L46

• Horizon API
  • Orderbook and Offers
    • order book summary view tests
    • fix "offers for account" bug
  • Effects Import
    • Interpret path payments, manage offers, and create passive offer operations
      for "trade" effects
    • Import effects from manage offer operations
    • Import effects from create passive offer operations
    • Fill out effects that occur due to Set Options effects
    • Add "raw xdr" storage to transaction entries in history db
  • Pathfinding
    • design endpoint
    • Port ruby prototype to go
    • Write test suite
  • Trading API
    • Add "trades" endpoint

After a transaction is created and the hash is given back to the client, getting the transaction with the hash results in a 404 if done too quickly.

Sometimes, uri template links returned have a slash right before query param. Filling in these parameters programmatically will create an invalid path.

Here is an example:

$  curl https://horizon-testnet.stellar.org/accounts/gcEuhxySh58bKtCY3UPaWQDR7a1BzGB3ePdxc4UrinkBJyxESe
{
  "_links": {
    "effects": {
      "href": "/accounts/gcEuhxySh58bKtCY3UPaWQDR7a1BzGB3ePdxc4UrinkBJyxESe/effects/{?cursor,limit,order}",
      "templated": true
    },
    "offers": {
      "href": "/accounts/gcEuhxySh58bKtCY3UPaWQDR7a1BzGB3ePdxc4UrinkBJyxESe/offers/{?cursor,limit,order}",
      "templated": true
    },
    "operations": {
      "href": "/accounts/gcEuhxySh58bKtCY3UPaWQDR7a1BzGB3ePdxc4UrinkBJyxESe/operations/{?cursor,limit,order}",
      "templated": true
    },
    "self": {
      "href": "/accounts/gcEuhxySh58bKtCY3UPaWQDR7a1BzGB3ePdxc4UrinkBJyxESe"
    },
    "transactions": {
      "href": "/accounts/gcEuhxySh58bKtCY3UPaWQDR7a1BzGB3ePdxc4UrinkBJyxESe/transactions/{?cursor,limit,order}",
      "templated": true
    }
  },
  "id": "gcEuhxySh58bKtCY3UPaWQDR7a1BzGB3ePdxc4UrinkBJyxESe",
  "paging_token": "77309415424",
  "address": "gcEuhxySh58bKtCY3UPaWQDR7a1BzGB3ePdxc4UrinkBJyxESe",
  "sequence": 77309411367,
  "balances": [
    {
      "currency_type": "native",
      "balance": 99960999999610
    }
  ]
}

Wrong (example shortened for clarity):

"href": "/accounts/abc/offers/{?cursor,limit,order}",

Correct:

"href": "/accounts/abc/offers{?cursor,limit,order}",

Moved from: stellar/js-stellar-base#25

This query would allow someone who wishes to place a market order of a given volume to understand the cost they will pay.

Input: normal orderbook args, and volume to sell
Output: a set of pricelevels where volume available at that price level is less than the target volume

This results in invalid next urls that look like:

href: '?order=asc&limit=10&cursor=46669114642432'

{
  "_links": {
    "effects": {
      "href": "/operations/136025909235713/effects/{?cursor,limit,order}",
      "templated": true
    },
    "precedes": {
      "href": "/operations?cursor=136025909235713\u0026order=asc"
    },
    "self": {
      "href": "/operations/136025909235713"
    },
    "succeeds": {
      "href": "/operations?cursor=136025909235713\u0026order=desc"
    },
    "transaction": {
      "href": "/transactions/136025909235712"
    }
  },
  "amount": "0.000001",
  "asset_type": "native",
  "from": "GBXEFLUSVV6PWKGIKEOBVO66EMMSPIG4LRUEGQEDUC6U7JGU4OHYR65X",
  "id": 136025909235713,
  "paging_token": "136025909235713",
  "to": "GBCR5OVQ54S2EKHLBZMK6VYMTXZHXN3T45Y6PRX4PX4FXDMJJGY4FD42",
  "type": 1,
  "type_s": "payment"
}

But hex IDs are used in /transaction endpoints (ex. /transactions/2a2beb163e2c68bd2377aab243d68225626d70263444a85556ec7271d4e46e03).

The endpoint to get the offers for a given an account is now broken:

ERRO[4818] sql: Scan error on column index 11: converting string "0.001" to a int64: strconv.ParseInt: parsing "0.001": invalid syntax  file=src/github.com/stellar/horizon/render/problem/main.go line=71

stellar=# select * from offers;
                         sellerid                         | offerid | sellingassettype | sellingassetcode |                      sellingissuer                       | buyingassettype | buyingassetcode |                       buyingissuer                       | amount | pricen | priced | price | flags | lastmodified 
----------------------------------------------------------+---------+------------------+------------------+----------------------------------------------------------+-----------------+-----------------+----------------------------------------------------------+--------+--------+--------+-------+-------+--------------
 GAKGBVPUDQAFQWZAUMSOSABSVCF2RYTUMGBNSEN4YPPD3OK5INNC5CYV |       4 |                1 | GBP              | GAGL56EZMYW2NXG3EEU3WGOUKRCL5LDYYPJ3T5FNP4AZ6QPKI532PTXN |               2 | US1234567890    | GAGL56EZMYW2NXG3EEU3WGOUKRCL5LDYYPJ3T5FNP4AZ6QPKI532PTXN |   4000 |      1 |   1000 | 0.001 |     0 |         1299

When a transaction fails, for instance when the offer in under funded or the reserve is to too low for changing trust, getting the transaction by its hash return a 404. The only way to find out what's happened is to dig into the txhistory table of the stellar db and decode the txresult.

This will allow someone to, for example, build a transaction history with age display without having to load date data afterwards

Should instead be a 400-class error specifically mentioning that the XDR was malformed.

In query_order_book_summary.go, getting the order book is in 2 phases, first the bid, then the ask.
getting the bid and ask array must be done in the same query, otherwise, under heavy load, one might get a bid than is greater than the ask.

Notes

• severity based logging
• can interface with external packages
• based on logrus
• logs SQL executions in debug log level
• dynamic log level settings

Crazy Ideas

• /logs/ admin endpoint that streams log data over SSE, optionally upping the log level while connected

from the HTTP protocol specification:

"The origin server MUST create the resource before returning the 201 status code. If the action cannot be carried out immediately, the server SHOULD respond with 202 (Accepted) response instead."

Since the account is not created until the transaction is applied, and a call to /accounts/ with the address will fail with 404 not found immediately after, the server should respond with a 202 status for friendbot.

When listing the operations for an account, there is no asset information regarding the manage_offer type, only the amount and the price are listed. Would it be possible to add the buying and selling asset ?

     {
        "_links": {
          "effects": {
            "href": "/operations/115964121088/effects/{?cursor,limit,order}",
            "templated": true
          },
          "precedes": {
            "href": "/operations?cursor=115964121088\u0026order=asc"
          },
          "self": {
            "href": "/operations/115964121088"
          },
          "succeeds": {
            "href": "/operations?cursor=115964121088\u0026order=desc"
          },
          "transactions": {
            "href": "/transactions/115964121088"
          }
        },
        "amount": 3,
        "id": 115964121088,
        "offer_id": 0,
        "paging_token": "115964121088",
        "price": {
          "d": 1,
          "n": 10
        },
        "type": 3,
        "type_s": "manage_offer"
      }

• When I connect to this endpoint, it returns the first 10 transactions and then my onerror callback gets called.
• the "after" parameter doesn't seem to be respected

Running scripts/run_tests.bash asks for a password, is it intended ?

On numerous occasion but not always, the following error occurs when running horizon-importer from a fresh db. To overcome this problem, drop the db again and restart.

History::Ledger Load (0.4ms)  SELECT  "history_ledgers".* FROM "history_ledgers"  ORDER BY "history_ledgers"."sequence" DESC LIMIT 1
  StellarCore::LedgerHeader Load (0.3ms)  SELECT  "ledgerheaders".* FROM "ledgerheaders" WHERE "ledgerheaders"."ledgerseq" = $1  ORDER BY "ledgerheaders"."ledgerhash" ASC LIMIT 1  [["ledgerseq", 1]]
  StellarCore::LedgerHeader Load (0.2ms)  SELECT  "ledgerheaders".* FROM "ledgerheaders" WHERE "ledgerheaders"."ledgerseq" = $1  ORDER BY "ledgerheaders"."ledgerhash" ASC LIMIT 1  [["ledgerseq", 1]]
  StellarCore::Transaction Load (0.2ms)  SELECT "txhistory".* FROM "txhistory" WHERE "txhistory"."ledgerseq" = $1  [["ledgerseq", 1]]
   (0.1ms)  BEGIN
  History::Account Exists (0.4ms)  SELECT  1 AS one FROM "history_accounts" WHERE "history_accounts"."id" = 0 LIMIT 1
  History::Account Exists (0.4ms)  SELECT  1 AS one FROM "history_accounts" WHERE "history_accounts"."address" = 'GCEZWKCA5VLDNRLN3RPRJMRZOX3Z6G5CHCGSNFHEYVXM3XOJMDS674JZ' LIMIT 1
   (0.2ms)  ROLLBACK
Actor crashed!
ActiveRecord::RecordInvalid: Validation failed: Id has already been taken, Address has already been taken
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/validations.rb:79:in `raise_record_invalid'
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/validations.rb:43:in `save!'
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/transactions.rb:291:in `block in save!'
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/transactions.rb:347:in `block in with_transaction_returning_status'
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:188:in `within_new_transaction'
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/transactions.rb:220:in `transaction'
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/transactions.rb:344:in `with_transaction_returning_status'
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/transactions.rb:291:in `save!'
    /var/lib/gems/2.2.0/gems/activerecord-4.2.1/lib/active_record/persistence.rb:51:in `create!'
    /horizon/app/jobs/history/ledger_importer_job.rb:471:in `create_master_history_account!'
    /horizon/app/jobs/history/ledger_importer_job.rb:19:in `block in perform'

The go-horizon server doesn't add the CORS headers to the client when it sends the http options verb with the Access-Control-Request-Headers and "Access-Control-Request-Method http header

$ curl -v http://localhost:8000/ -H "Origin: demo.com" -X OPTIONS -H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers:accept, authorization"
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> OPTIONS / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8000
> Accept: */*
> Origin: demo.com
> Access-Control-Request-Method: GET
> Access-Control-Request-Headers:accept, authorization
> 
< HTTP/1.1 200 OK
< Date: Wed, 19 Aug 2015 10:10:01 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact

It works fine for a standard http get:

$ curl -v http://localhost:8000/ -H "Origin: demo.com"
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8000
> Accept: */*
> Origin: demo.com
>
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: demo.com
< Content-Type: application/hal+json
< Vary: Origin
< X-Ratelimit-Limit: 3600
< X-Ratelimit-Remaining: 3597
< X-Ratelimit-Reset: 2924
< Date: Wed, 19 Aug 2015 09:18:59 GMT
< Content-Length: 759

I'm opening this issue to discuss an implementation problem in the open and hopefully enlist some help from the community. Let me first begin by describing the feature at hand then discuss the possible solutions I see.

Feature: Trades for order book API endpoint

This feature will provide a pageable list of all exchanges of value ("trades") that occur between a pair of assets (the "order book"). These trades are created when offers cross (triggered from ManageOffer or CreatePassiveOffer operations) or when an account makes a PathPayment.

Trades don't concretely exist within data model of stellar-core, but having a reified trade within horizon is very useful; Having it provides a straightforward implementation for any analysis on the history of an order book as well as low-cost interfaces to the distributed exchange provided by the Stellar network.

Current situation

Presently, trades are stored as a sub-type of "Effect". Effects are concretely tied to an individual accounts within the Stellar network, with any attributes unique to a sub-type of effect stored within a jsonb column. This leads to the current roadblock for implementing the "Trades for orderbook" endpoint: The information that identifies what order book a trade took place on is in the jsonb "details" column.

We must find a way to efficiently query for trade effects by order book.

Solution 1: Partial Indexing

By using postgres partial indexes we could index just the trade effects, ignoring all the other types. This seems like a pretty clean solution.

Drawbacks: I don't have an operational experience with partial indexes or jsonb indexes... there may be tricks and traps that we do not realize.

Solution 2: A new table

For this solution, we would create a new table (probably named "trades") that contains one row per trade and has indexed columns to service the query used to drive the "trades for orderbook" query.

Drawbacks: Storage. The physical size of the horizon database would grow more quickly as we would be duplicating data and that extra size would be larger than just the size of the extra index as described in solution 1.

Solution 3: ???????

What else do we have, any contributions?

Personally, I'm leaning towards solution 1, but would love to hear any and all feedback about additional pros/cons of each proposed solution.

The present system for retrieving data from the query system db.Results() and db.First() are useful for the streaming system, where we can't easily update a struct provided by the caller, but in the general case it is cumbersome, involving 2 error checks:

result, err := First(q)
// error check the db call
account, ok := result.(AccountRecord)
// error check the type assertion

A better pattern, used by sqlx for example is to populate a provided struct:

var result AccountRecord
err := Get(q, &result)
// error check the query

We should provide the same helpers for the go-horizon/db package