Code Monkey home page Code Monkey logo

cosmossqlcalipers's Introduction

CosmosSQLCalipers

CosmosSQLCalipers is a basic Cosmos SQL API benchmarking utility. It enables developers to model and understand the impact of the following parameters:

  • Document sizes
  • Partition key based SELECT queries vs point reads
  • Impact of sync versus async APIs
  • Request unit (RUs) consumption
  • Network latencies and throughput
  • Supports masterkey and SPN based authentication

This enables developers to get a preview into the overall scalability, response times and cost considerations when evaluating Cosmos SQL API.

Use Azure SDK v4

Project upgraded to v4.33.1. Following consistency levels are running fine:

  • STRONG
  • BOUNDED_STALENESS
  • SESSION
  • CONSISTENCY_PREFIX
  • EVENTUAL

Overview

The utility executes the following workflow:

  • Provisions a collection based on input parameters
  • Allocates the provisioned RUs
  • Creates and inserts documents into the collection
  • Executes the query workload based on the operation specified in the input parameters
  • Deletes the documents explicitly at the end of the exercise

Indexing policy has a significant impact on RU consumption. So for the sake of simplicity, CosmosSQLCalipers deploys a single index for partition key lookups. Collected metrics include:

  • Throughput
  • Network latencies
  • RUs consumed

Operations that can be exercised using this tool includes:

  1. SQL_ASYNC_PARTITION_KEY_READ: Executes partition key based queries asynchronously
  2. SQL_SYNC_PARTITION_KEY_READ: Executes partition key based queries synchronously
  3. SQL_ASYNC_POINT_READ: Executes point read operations asynchronously
  4. SQL_SYNC_POINT_READ: Executes point read operations synchronously
  5. SQL_ASYNC_UPSERT: Executes upsert operations asynchronously
  6. SQL_SYNC_UPSERT: Executes upsert operations synchronously
  7. SQL_ASYNC_REPLACE: Executes replace operations asynchronously
  8. SQL_SYNC_REPLACE: Executes replace operations synchronously
  9. ALL_SYNC_OPS: Executes all synchronous operations serially. The execution cycle starts with partition key based reads followed by point reads.
  10. ALL_ASYNC_OPS: Executes all asynchronous operations serially. The execution cycle starts with partition key based reads followed by point reads.
  11. SQL_ALL: Executes async partition key read, sync partition key read, async point read and sync point read operations serially.

SQL API workflows executed

  1. SQL_ASYNC_PARTITION_KEY_READ and SQL_SYNC_PARTITION_KEY_READ options
    1. createItem()
    2. queryItem()
    3. deleteItem()
  2. SQL_ASYNC_POINT_READ and SQL_SYNC_POINT_READ options
    1. createItem()
    2. readItem()
    3. deleteItem()
  3. SQL_ASYNC_READ_ALL_ITEMS and SQL_SYNC_READ_ALL_ITEMS options
    1. createItem()
    2. readAllItems()
    3. deleteItem()
  4. SQL_ASYNC_UPSERT and SQL_SYNC_UPSERT options
    1. createItem()
    2. upsertItem()
    3. deleteItem()
  5. SQL_ASYNC_REPLACE and SQL_SYNC_REPLACE options
    1. createItem()
    2. replaceItem()
    3. deleteItem()
  6. ALL_SYNC_OPS and ALL_ASYNC_OPS options
    1. createItem()
    2. queryItem()
    3. readItem()
    4. readAllItems()
    5. upsertItem()
    6. replaceItem()
    7. deleteItem()
  7. SQL_ALL
    1. Invokes all async and sync ops

Instructions

  1. Create a Cosmos DB SQL API account
    1. Assign consistency level you want to test with
    2. Select region depending on your deployment preferences
    3. Single or multiple regions
    4. Choose an active/active or active/passive topology
  2. Run the benchmark test against the provisioned Cosmos account.

Program arguments

Execute Cosmos Benchmark utility using two different modes now, one with SPN based authentication and other with Master key based authentication

Usage: cosmosbenchmark [COMMAND]
Commands:
  spn        Azure SPN based authentication test
  masterkey  Azure master key based authentication test

Usage command for 'spn'

Usage: cosmosbenchmark spn [-dc] -c=<Collection Name> -cl=<Client Id>
                           -cs=<Client Secret> -d=<Database Name> -e=https:
                           //<ENDPOINT>.documents.azure.com:443/
                           [-l=<consistencyLevel>] -n=<Number of documents>
                           [-o=<operation>] -p=<Payload Size in KBs>
                           [-r=<Maximum retry attempts>] [-rf=<reporter>]
                           -rg=<resource group> [-ru=<Provisioned RU's>]
                           [-rwi=<Retry wait time in sec.>]
                           -sub=<subscriptionId> -t=<Tenant Id>
Azure SPN based authentication test
  -c, --collection=<Collection Name>
         Cosmos collection name
      -cl, --clientId=<Client Id>
         Client Id
      -cs, --clientSecret=<Client Secret>
         Client Secret
  -d, --database=<Database Name>
         Cosmos Database Name
      -dc, --deleteContainer
         Delete and recreate the container (true/false), defaults to true
  -e, --endpoint=https://<ENDPOINT>.documents.azure.com:443/
         Cosmos Service endpoint
  -l, --consistency=<consistencyLevel>
         Consistency level to be used (Strong, BoundedStaleness, Session,
           Eventual, ConsistentPrefix). If omitted the Session consistency
           would be used and should be weaker than account level consistency
  -n, --numberofdocument=<Number of documents>
         Number of documents to be tested
  -o, --operation=<operation>
         Primary operation being used (SQL_ASYNC_PARTITION_KEY_READ,
           SQL_SYNC_PARTITION_KEY_READ, SQL_SYNC_POINT_READ,
           SQL_ASYNC_POINT_READ, SQL_ALL, ALL_SYNC_OPS, ALL_ASYNC_OPS,
           SQL_SYNC_UPSERT, SQL_ASYNC_UPSERT, SQL_SYNC_REPLACE,
           SQL_ASYNC_REPLACE, SQL_SYNC_READ_ALL_ITEMS,
           SQL_ASYNC_READ_ALL_ITEMS, SQL_SYNC_DELETE, SQL_ASYNC_DELETE). If
           omitted the SQL_ALL operation would be used.
  -p, --payloadsize=<Payload Size in bytes>
         Document size
  -r, --retry=<Maximum retry attempts>
         Number of retry attempts
      -rf, --reporting=<reporter>
         Reporting format to be used (CONSOLE, CSV). If omitted the CONSOLE
           operation would be used.
      -rg, --resourceGroup=<resource group>
         Resource group for the Cosmos account
      -ru, --provisionedRUs=<Provisioned RU's>
         RUs to be provisioned when Cosmos container is created
      -rwi, --retry-wait-interval=<Retry wait time in sec.>
         max retry wait interval (in seconds)
      -sub, --subscriptionId=<subscriptionId>
         Subscription Id to be used
  -t, --tenantId=<Tenant Id>
         Tenant Id

Usage command for 'masterkey'

Usage: cosmosbenchmark masterkey [-dc] -c=<Collection Name> -d=<Database
                                 Name> -e=https://<ENDPOINT>.documents.azure.
                                 com:443/ -k=<Access Key>
                                 [-l=<consistencyLevel>] -n=<Number of
                                 documents> [-o=<operation>] -p=<Payload Size
                                 in KBs> [-r=<Maximum retry attempts>]
                                 [-rf=<reporter>] [-ru=<Provisioned RU's>]
                                 [-rwi=<Retry wait time in sec.>]
Azure master key based authentication test
  -c, --collection=<Collection Name>
                           Cosmos collection name
  -d, --database=<Database Name>
                           Cosmos Database Name
      -dc, --deleteContainer
                           Delete and recreate the container (true/false),
                             defaults to true
  -e, --endpoint=https://<ENDPOINT>.documents.azure.com:443/
                           Cosmos Service endpoint
  -k, --key=<Access Key>   Access key
  -l, --consistency=<consistencyLevel>
                           Consistency level to be used (Strong,
                             BoundedStaleness, Session, Eventual,
                             ConsistentPrefix). If omitted the Session
                             consistency would be used.
  -n, --numberofdocument=<Number of documents>
                           Number of documents to be tested
  -o, --operation=<operation>
                           Primary operation being used
                             (SQL_ASYNC_PARTITION_KEY_READ,
                             SQL_SYNC_PARTITION_KEY_READ, SQL_SYNC_POINT_READ,
                             SQL_ASYNC_POINT_READ, SQL_ALL, ALL_SYNC_OPS,
                             ALL_ASYNC_OPS, SQL_SYNC_UPSERT, SQL_ASYNC_UPSERT,
                             SQL_SYNC_REPLACE, SQL_ASYNC_REPLACE,
                             SQL_SYNC_READ_ALL_ITEMS, SQL_ASYNC_READ_ALL_ITEMS,
                             SQL_SYNC_DELETE, SQL_ASYNC_DELETE). If omitted the
                             SQL_ALL operation would be used.
  -p, --payloadsize=<Payload Size in KBs>
                           Document size
  -r, --retry=<Maximum retry attempts>
                           Number of retry attempts
      -rf, --reporting=<reporter>
                           Reporting format to be used (CONSOLE, CSV). If
                             omitted the CONSOLE operation would be used.
      -ru, --provisionedRUs=<Provisioned RU's>
                           RUs to be provisioned when Cosmos container is
                             created
      -rwi, --retry-wait-interval=<Retry wait time in sec.>
                           max retry wait interval (in seconds)

Building and running

mvn clean package

Executing using the 'masterkey' option

mvn exec:java -Dexec.mainClass="com.cosmoscalipers.Benchmark" -Dexec.cleanupDaemonThreads=false -Dexec.args="masterkey --endpoint=<cosmos-endpoint> --database=<database-name> --collection=<container-name> --consistency=SESSION --numberofdocument=1 --payloadsize=1 --key=<access key for the account>"

Executing using the 'spn' option

Following are few pre-requisites before you execute the application

  • The Service Principal needs to be created beforehand and make sure it has been assigned "Cosmos DB Operator" role on the Cosmos account
  • In order for the CRUD operations to work on the Cosmos database, you would need to assign the required roles to the service principal, the details of which are available here
mvn exec:java -Dexec.mainClass="com.cosmoscalipers.Benchmark" -Dexec.cleanupDaemonThreads=false -Dexec.args="spn --subscriptionId=<subscriptionId> --resourceGroup=<resourceGroup> --clientId=<client-id> --clientSecret=<client-secret> --tenantId=<tenant-id> --endpoint=<cosmos-endpoint> --database=<database-name> --collection=<container-name> --consistency=SESSION --numberofdocument=100 --payloadsize=1 --operation SQL_ASYNC_POINT_READ"

cosmossqlcalipers's People

Contributors

deepub avatar faizmsft avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.