Code Monkey home page Code Monkey logo

dynamodb.sql's Introduction

Issue Stats Issue Stats

DynamoDb.SQL (@DynamoDbSQL)

This library gives you the capability to execute query and scan operations against Amazon DynamoDB using a SQL-like syntax by extending the existing functionalities of AmazonDynamoDBClient and DynamoDBContext classes found in the standard .Net AWS SDK.

This guide contains the following sections:

  • Basics - how to get started using this library
  • Features - what you can do with this library
  • Examples - links to examples in F# and C#

You can view the release notes here, and please report any issues here.

NuGet Status

Basics

Getting Started

Download and install DynamoDb.SQL using NuGet.

NuGet package

Now suppose we have a DynamoDB tabled called GameScores like the following:

!table

To find all the scores for the player with UserId "theburningmonk-1" we can simply execute the query like the following:

open Amazon.DynamoDBv2
open Amazon.DynamoDBv2.DataModel

let awsKey      = "MY-AWS-KEY"
let awsSecret   = "MY-AWS-SECRET"
let region      = RegionEndpoint.USEast1

let client = new AmazonDynamoDBClient(awsKey, awsSecret, region)
let ctx    = new DynamoDBContext(client)

let query  = "SELECT * FROM GameScores WHERE UserId = \"theburningmonk-1\""
let scores = ctx.ExecQuery<GameScore>(selectQuery)

whilst the above example is in F#, the same extension methods are accessible from C# too, check out the full range of examples in the Examples section.

For a detailed run-down of the syntax please refer to this page.

Features

Simple SQL-like syntax

This library lets you use a SQL-like syntax for performing query and scan operations against DynamoDB. If you're new to DynamoDB and is not clear on the difference between the two, please refer to the DynamoDB documentation here and guidelines here.

Using the appropriate extension methods on AmazonDynamoDBClient and DynamoDBContext you will be able to query/scan your DynamoDB table like this:

let selectQuery = "SELECT * FROM GameScores WHERE UserId = \"theburningmonk-1\""
let response    = client.Query(selectQuery)

Whilst the syntax for both query and scan operations are similar, there are minor differences and some comparisons (such as CONTAINS and IN (...)) are only supported in scan operations by DynamoDB.

For a detailed run-down of the syntax please refer to this page.

Count-only queries

If you only want to find out the number of items that matches some conditions and not the items themselves, then you can save yourself some bandwidth and read capacity units by using a COUNT query:

let countQuery    = "COUNT * FROM GameScores WHERE UserId = \"theburningmonk-1\""
let countResponse = client.Query(countQuery)

Note: count queries are only supported when you're working with the AmazonDynamoDBCient.

Selecting specific attributes

If you only want to retrieve certain attributes from your query, then you can save yourself some bandwidth and potentially processing power by specifying exactly which attributes you want to retrieve:

let selectQuery = "SELECT UserId, GameTitle, Wins FROM GameScores WHERE UserId = \"theburningmonk-1\""
let response    = client.Query(selectQuery)

Ordering and Limiting

Often you will want to retrieve only the top or bottom X number of items based on the natural sorting order of the range key values. Therefore it often makes sense to combine the ORDER and LIMIT clauses in your query.

For example, in our GameScores table (see above), to find the top 3 scoring games for a given user we can write:

let selectQuery = "SELECT * FROM GameScores WHERE UserId = \"theburningmonk-1\" ORDER DESC LIMIT 3"
let response    = client.Query(selectQuery)

Note: in DynamoDB, you can only order on the range key values hence why there's no option for you to specify what column to order on in the query syntax.

Throttling

As stated in the DynamoDB best practices guide, you should avoid sudden bursts of read activity.

To throttle your query or scan operation, you can use the PageSize option in your query to throttle the amount of read capacity units that your query consumes in one go:

let selectQuery = "SELECT * FROM GameScores WHERE UserId = \"theburningmonk-1\" WITH (PageSize(10))"
let response    = client.Query(selectQuery)

this query will fetch 10 results at a time, if there are more than 10 results available then additional requests will be made behind the scene until all available results have been retrieved.

Note: using the PageSize option means your query or scan will take longer to complete and require more individual requests to DynamoDB.

Parallel scans

If your table is sufficiently large (DynamoDB documentations suggests 20GB or larger), it's recommended that you take advantage of the parallel scans support in DynamoDB to speed up the scan operations.

To use parallel scans, you can use the Segments option in your scan query:

let selectQuery = "SELECT * FROM GameScores WHERE GameTitle = \"Starship X\" WITH (Segments(10))"
let response    = client.Scan(selectQuery)

this query will make ten parallel scan requests against DynamoDB and the operation will complete when all ten 'segments' have completed and returned all their results.

Note: using parallel scan will consume large amounts of read capacity units in a short burst, so you should plan ahead and up the throughput of your table accordingly before starting the parallel scan!

Local Secondary Index support

AWS announced support for Local Secondary Indexes on April 18, 2013, for more details please refer to the DynamoDB documentations page here and guidelines for using Local Secondary Indexes.

Support for local secondary index is available since version 1.2.1 using the INDEX option inside the WITH clause.

For example, suppose the aforementioned GameScores table has a local secondary index called TopScoreIndex:

table-indexes

We can query the table using this index and optionally specify whether to retrieve all attributes or just the attributes projected into the index (any attributes that are not on the index will be retrieved from the table using extra read capacity):

let selectQuery = "SELECT * FROM GameScores 
                   WHERE UserId = \"theburningmonk-1\" 
                   AND TopScore >= 1000 
                   WITH(Index(TopScoreIndex, true))"
let response = client.Query(selectQuery)

For more details, please read this post.

Global Secondary Indexes support

AWS announced support for Global Secondary Indexes on December 12, 2013, for more details please refer to the DynamoDB documentations page here and guidelines for using Global Secondary Indexes.

Global Secondary Indexes, or GSI is supported through the same INDEX option as local secondary index above, the index name specified in the INDEX option can be any local or global index on the table.

For example, to query against the global secondary index GameTitleIndex on our GameScores table (see above):

let selectQuery = "SELECT * FROM GameScores 
                   WHERE GameTitle = \"Starship X\" 
                   AND TopScore >= 1000
                   WITH(Index(GameTitleIndex, false), NoConsistentRead)"
let response = client.Query(selectQuery)

Important: although the queries look identical, compared to local secondary indexes there are a couple of key differences you need to be aware of when querying against global secondary indexes:

  • you must add the NoConsistentRead option in your query as global secondary indexes only support eventually consistent reads, if you try to do a consistent read against a global secondary index it will result in an error from DynamoDB;
  • when you created the global secondary index, if you didn't choose All Attributes as the Projected Attributes for the index, then you must set the "all attributes" flag in the Index option to false (i.e. Index(IndexName, false))

Examples

Here's a handful of examples in C# and F#, feel free to check out the respective project under the examples folder too, it also contains F# script to create the sample table and seeding it with test data you need to run these examples.

Query

Get all rows for a hash key C# F#
Query with range key C# F#
Query with ORDER and LIMIT C# F#
Disable consistent read C# F#
Throttling C# F#
Selecting specific attributes C# F#
Query with Local Secondary Index (all attributes) C# F#
Query with Local Secondary Index (projected attributes) C# F#
Query with Global Secondary Index (projected attributes) C# F#

Scan

Basic scan C# F#
Scan with LIMIT C# F#
Throttling C# F#
Selecting specific attributes C# F#
Parallel scans C# F#
Disable returning consumed capacity C# F#

Maintainer(s)

The default maintainer account for projects under "fsprojects" is @fsprojectsgit - F# Community Project Incubation Space (repo management)

dynamodb.sql's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dynamodb.sql's Issues

Parser error when any field contains underscore in it's name

Error in Ln: 1 Col: 59
es_active WHERE day = "2021/04/29" AND create_date_device BEGINS WITH "2021/04/
^

The parser backtracked after:
Error in Ln: 1 Col: 65
ve WHERE day = "2021/04/29" AND create_date_device BEGINS WITH "2021/04/29" W
^
Expecting: '<', '<=', '=', '>', '>=' or 'begins with' (case-insensitive)

The parser backtracked after:
Error in Ln: 1 Col: 65
ve WHERE day = "2021/04/29" AND create_date_device BEGINS WITH "2021/04/29" W
^
Expecting: 'between' (case-insensitive)
: InvalidQueryException
at DynamoDb.SQL.Parser.parseDynamoQuery(String str)

Querying on index with very long number

Hi Yan,
This could be the same as I mentioned before which you fixed it with long numbers. This time when I'm querying on index with a long number then it is not returning any results and with small number it does. I debugged it a little and found that it is jumping one number up. Following is the query I'm trying and the snapshot of the converted filter

This works
SELECT * FROM pd_cs_friendshiprequest WHERE ForUserId = 578318698 WITH(Index(ForUserIdIndex, false), NoConsistentRead)

This doesn't work
SELECT * FROM pd_cs_friendshiprequest WHERE ForUserId = 10205471065416239 WITH(Index(ForUserIdIndex, false), NoConsistentRead)

http://screencast.com/t/Ti4tizS51

I guess, it will be again a simple fix while converting string to number.
Hoping to hear from you soon.

Move to new AWSSDK

  • investigate compatibility issue with having both AWSSDK.dll and AWSSDK.DynamoDB.dll loaded

Join example missing

Table Join Example missing.

Please kindly provide some examples of table joins (join, left join)

DynamoDb.SQL.InvalidQueryException when table names have '_' underscore characters

Hi there,
I'm getting DynamoDb.SQL.InvalidQueryException when table name has underscores in it. It works fine for other tables without such character. I'm not able to find where it is check for underscores in the table names but it is definitly splitting on the bases of it. (Can't understand F# Code).

Can you let me know how to fix this? Is there any other limitation in table names?

Error on queries on tables with special characters in their names

I tried querying a table with a "." in it's name. I get the following error: "Expecting: 'where' (case-insensitive)". I get the same error when the table has a hyphen in its name.

I tried using brackets around the table name, as one would do in a SQL client. No luck. I get this error in that case: "Expecting: table name"

Long numbers are not working

Hi there again,
I have just noticed that the long numbers are not working in sql.
Consider the following two examples

// Not working even the data is present
SELECT * FROM pd_cs_chatroomuser WHERE UserId = 10205471065416056 WITH(Index(UserIdIndex, false), NoConsistentRead)

// Working
SELECT * FROM pd_cs_chatroomuser WHERE UserId = 10205471065 WITH(Index(UserIdIndex, false), NoConsistentRead)

ILMerge dependencies into assembly

It would be good if you could use ILMerge to merge the dependencies into the DynamoDB.SQL assembly. That way, we could use any version of AWSSDK.dll along side with any version of DynamoDB.SQL.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.