tarantool / tt Goto Github PK
View Code? Open in Web Editor NEWCommand-line utility to manage Tarantool applications
License: Other
Command-line utility to manage Tarantool applications
License: Other
tt connect [url|instance_name]
Connect to remote tarantool or tarantool instance (for local).
If there is only one local instance is present, You can use tt connect
without additional arguments.
-x lua|sql
- request format-I
- non interactive mode (example: echo _G| tt connect -I
)-i file
- pass the file (in non-interactive mode. See tarantoolctl eval
)see tarantoolctl rocks
The module will be used to build an application for local development.
Implement the logrotate
module similary to tarantoolctl logrotate
.
The case is similar to #44 .
The module provides integration with bash/zsh completion system.
tt completion argument[,argument...]
suggests about tt argument[,argument...]
Now when restarting the instance, the old configuration will be used, but we can change some settings (like restarting) and we don't want to force a restart of the instance to apply them. The suggestion is to add a re-read of the configuration when the instance is restarted.
see cartridge create
To specify the location of wal
, snapshots
(something else) needs to implement a box.cfg()
wrapper, like in the tarantoolctl
.
Pack application into a distributable bundle
see cartridge pack
Like #24 but import/export using crud.
The module
tvisord.auth
contains
Commands
$ tt tvisordauth --method
basic
$ tt tvisordauth --check "basic dXNlcjpwYXNzd29yZA=="
ok
If authorisation is not configured, then --method
returns disabled
, --check
returns ok
for every request.
Implement installation opensource versions of tt
and tarantool
from the github repository.
The commands tt install tt
and tt install tarantool
must clone the repository from github and install the binaries to the bindir
of environment.
The user can specify the version of the product they want like tt install tarantool=2.8
.
The current tarantool
is specified by using a symlink tarantool
in environment bindir
. Binares is stored in bindir
with specified versions in the filenames.
To upgrade to an already built version, the tt install
command is also used. To reinstall already built binaries, the --reinstall
flag is used.
prints current CLI version
$ tt version
Tarantool CLI version 1.2.3, linux/amd64. commit: 5770f5e
$ tt version --short
1.2.3
$ tt version --short --commit
1.2.3.5770f5e
The module will be used to pack application into a distributable bundle.
As an operations engineer, I want to be able to easily collect instance metrics through the CLI. I need this to diagnose and contact support. I want to collect metrics from my work machine without having to shell the node running the Tarantool instance.
Acceptance Criteria:
Offer first public version of tt that would look modern, convenient and wishful.
Here we leave tarantooctl cat
/ tarantoolctl play
out of the scope: it does not look as something that is often used.
Sum: 62sp.
Continuation of work on issue #34
Modify commands to support cartridge-cli application:
Now tt
can write the log to a specified file and rotate it according to settings.
It is required to implement the ability to generate logs according to the syslog standard.
In integration tests we have hardcoded ports:
Example
This can cause tests to fail:
def test_play_test_remote_instance(tt_cmd, tmpdir):
# Copy the .xlog and instance config files to the "run" directory.
test_app_path = os.path.join(os.path.dirname(__file__), "test_file")
shutil.copy(test_app_path + "/test.xlog", tmpdir)
shutil.copy(test_app_path + "/remote_instance_cfg.lua", tmpdir)
# Play .xlog file to the remote instance.
cmd = [tt_cmd, "play", "localhost:3301", "test.xlog", "--space=999"]
instance = subprocess.Popen(["tarantool", "remote_instance_cfg.lua"], cwd=tmpdir)
# The delay is needed so that the instance has time to start and configure itself
sleep(1)
rc, output = run_command_and_get_output(cmd, cwd=tmpdir)
instance.kill()
> assert rc == 0
E assert 1 == 0
test/integration/play/test_play.py:51: AssertionError
It is proposed to create a helper that looks for a free TCP port. This will greatly reduce the chance of falsely failed tests.
The module is used to start (not use) supervisor daemon.
tt tvisord [supervisors options]
Config options:
# tarantool.yaml
tvisord:
port: 3201 # default value
tls: # optional
cert: path/to
key: path/to
pid: path/to/tvisord.pid
log:
type: file # disabled, syslog
dir: path/to/logdir
Options:
--port
- listen port--tls{-cert,-key}
- TLS optionsHow to work:
tvisord.port
Request:
POST /tt/etc/tarantool HTTP/1.0
Content-Type: application/json
{
"module": "version",
"root": "/etc/tarantool",
"args": [],
"opts": { "short": true }
}
Response:
200 Ok
Content-Type: application/json
{
"status": "ok",
"version": "2.3.4.5"
}
tt module
(using ENV=TVISORD=yaml
). Parse its output and response.TODO: To auth, the server uses tt tvisorauth
Update: see #23
Implement start
, stop
and status
commands for a local instance. The commands should replace the same from tarantoolctl
.
The instance must be started under a watchdog
, which can restart the instance (or no) in case of a crash (according to option in config). Interaction with the instance by signals must be done through the watchdog (for example, if we want to stop the instance, we should send the SIGTERM
signal to the watchdog).
To work with the "cartridge" we want to reuse the https://github.com/tarantool/cartridge-cli module, except the following commands:
and probably:
The following implementation is suggested:
Add "cartridge-cli" to "tt" as a submodule with some wrapper that will mask excluded commands and pass others. Exceptions must also be taken into account for the "help" command.
Commands:
I'm not so good in command-line interface description language, but seems that the generally accepted standard is not always followed in `tt help'.
Need checks this guess and fix the typos.
tt
tool, which should make it easy to extend its functionality by adding modules (in fact, almost all the functionality of the module will be implemented through modules).In the process of developing a module, it is needed to keep in mind the possibility of having 3 types of modules:
tt
Yet several points of implementation of the basic tt
:
(The description was changed by @LeonidVas )
Replace the connector used in module tt connect
from https://github.com/FZambia/tarantool to https://github.com/tarantool/go-tarantool/
run local tarantool
tt run
and tarantool
are equivalent, but
tt run
detects if it is a local starttt
binary, exec it if the binary is found and not executed yet (basic tt functional)tarantool
binary, exec it if the binary is found or exec system tarantoolImplement tt coredump
module to collect and inspect tarantools coredumps.
The following set of commands are suggested:
tt coredump pack
- to collect and pack tarballs with coredump with all unnecessary libs.tt coredump unpack
- to unpack the tarball.tt coredump inspect
- to research the coredump.The following scripts are used as the basis for developing this module: tarantool/tarantool@e733f08
tt run
should to have the same functionality as tt start
, except that it does not use watchdog
and output should not be redirected, just write to stdout
, stderr
. The main purpose of this command is to check during development how the application will work under tt
.
Also, in this mode. no one command of tt
which work through watchdog
can be used to interact with an instance.
The same as tarantoolctl cat
and tarantoolctl play
, but it has to run tt connect
for access to a node.
Now tt completion
is implemented for internal modules. Needs to propose and implement a mechanism for working tt completion
with external modules. I think all necessary information can be requested similar to help
and description
from module. For example, we can use the flag completion
for this.
In the light of upcoming rename of the tool to tt
, I suggest adding a fancy greeting which will be displayed in the logs when the tool starts or an interactive mode is entered:
.:-.o-.............```````................:/-`
:hdddddddhddhssssss+-..----/ooooooooossssssssyy`
+syyyhyyyyyyss//////////////////+//++shhhhhhhddd.
`sooo+++++++ooo++ysssssss+++++oooooooyhhhhhhddddh
oys+//oooooooso/shhhhhhhyyyyyyyyyyyyo+++++//+++:
`:s+odhhdhdhdm+sdy++ ``-+:
-oydmmmmmmmhshd+++ :`
`shmmmmmmmmsyhhsss+..--`
-ymmNmmmmmdy/-.......`
ohmNNmmmmmy/
-hmmNNNNNmds`
ohmNNNNNmmh+
-hmNNNNNNmdy.
odNNNNNNNmh/
:hdddmmmddh-
::`
The greeting will be toggleable in settings.
As was proposed by @Totktonada in his review, I made a list for the required follow-up activity related to both tarabrt.sh
and gdb.sh
introduced in scope of tarantool/tarantool#5569. Major changes relate to the case when debug info is stripped from the Tarantool binary (e.g. in CentOS 8 binary packages until tarantool/tarantool#4611 is resolved). The following enhancements should to be done for the tools:
-h
/ --help
)gdb.sh
user whether Tarantool binary is strippedgdb.sh
automaticallygdb.sh
output, when we see the stripped executable)
Examples:
$ curl -LfsS https://download.tarantool.org/tarantool/release/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm
$ mkdir tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
$ cd tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
$ rpm2cpio ../tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm | cpio -idmv
./usr/lib/debug
./usr/lib/debug/.build-id
./usr/lib/debug/.build-id/a3
./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60
./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60.debug
./usr/lib/debug/usr
./usr/lib/debug/usr/bin
./usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
64743 blocks
(All are symlinks except tarantool-2.8.0.0-1.el8.x86_64.debug.)
$ file usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter \004, for GNU/Linux 3.2.0, BuildID[sha1]=a3ad84db98a9b14e70d2915271904ddfe26b5f60, with debug_info, not stripped, too many notes (256)
tt import name -format csv -space users users.csv
tt import name -format csv -space users < file.csv
tt export name -format csv -space users users.csv
tt export name -format csv -space users > users.csv
Options:
--format
- file format (CSV is default)--space
- space name for import/export. Required option--match name1=csvrow1:name2=csvrow2:...
- Match for CSV and space fields for import and export.
--header=on|off
- (off
- default).
on
- export fieldnames to the first CSV line, import fieldnames from the first CSV line (can raise error)off
- the first CSV line is a data lineYou could make alias --match header
and --header on
.
Import (only) options:
--progress {file}
- use the file to store import progressIf the option "match
" option doesn't contain any primary key fields, the utility will generate a primary key for each record: autoincrement for numeric
fields, UUID-string for string
fields, UUID
for UUID
fields.
--failed {skip|error}
- behaviour with error tuples (example duplicate key or constraint errors)--update {new|all}
- update exists records or create new only. (The option means something if PK is present in input CSV)As an operations engineer, I want to be able to easily back up my instance data and configuration. I need this to reduce the load on the cluster when restoring an instance in the event of its death. I also need it to easily make a copy of the sales in a test environment.
I want to be able to back up my work machine without having to shell the node running the Tarantool instance. It is also important for me that the tools used and require the minimum possible additional configuration of the system in which it works.
Acceptance Criteria:
Implement the check
module similary to tarantoolctl check
.
As an operations engineer, I want to be able to easily back up my instance data and configuration. I need this to reduce the load on the cluster when restoring an instance in the event of its death. I want to be able to back up my work machine without having to shell the node running the Tarantool instance. It is also important for me that the tools used and require the minimum possible additional configuration of the system in which it works.
Acceptance Criteria:
The administrator can view the list of instances on a specific node (similar to CLI: get coredumps with request)
The administrator has the ability to deploy a backup of an instance on a specific node.
Before deploying an instance backup, the Tarantool version on the target node is checked. * Here we focus on the distribution version, not a separate component! *
Before deploying a backup of an instance on the target node, a check is made for the presence of this instance. Verification is performed by name or UUID. If the instance already exists, the administrator is asked to confirm that this instance will be replaced with a new one from the backup
The administrator has the ability to specify: expand only data or configuration + data.
As an Operations Engineer, I want to be able to easily collect debug information to troubleshoot crashes and forward to support. I want to collect debug information from my work machine without having to shell the node running the Tarantool instance.
Acceptance Criteria:
$ ps ax|grep tarantool
3967854 ? Sl 0:00 tarantool abc.lua -e --- This is a launch script that does the necessary preparation -- before launching an instance. -- The script is delivered inside the "tt" binary and is launched -- to execution via the `-e` flag when starting the application instance. -- AFAIU, due to such method of launching, we can reach the limit of the -- command line length ("ARG_MAX") and in this case we will have to create -- a file with the appropriate code. But, in the real world this limit is -- quite high (I looked at it on several machines - it equals 2097152) -- and we can not create a workaround for this situation yet. -- -- Several useful links: -- https://porkmail.org/era/unix/arg-max.html -- https://unix.stackexchange.com/a/120842 local os = require('os') local console = require('console') local log = require('log') local title = require('title') --- Start an Instance. The "init" file of the Instance passes -- throught "TT_CLI_INSTANCE". local function start_instance() local instance_path = os.getenv('TT_CLI_INSTANCE') if instance_path == nil then log.error('Failed to get instance path') os.exit(1) end title.update{ script_name = instance_path, __defer_update = true } -- Preparation of the "console" socket. local console_sock = os.getenv('TT_CLI_CONSOLE_SOCKET') if console_sock ~= nil and console_sock ~= '' then local cons_listen_sock = console.listen(console_sock) -- tarantool 1.10 does not have a trigger on terminate a process. -- So the socket will be closed automatically on termination and -- deleted from "running.go". if box.ctl.on_shutdown ~= nil then local close_sock_tr = box.ctl.on_shutdown(function() box.ctl.on_shutdown(nil, close_sock_tr) local res, err = pcall(cons_listen_sock.close, cons_listen_sock) if not res then log.error('Failed to close console socket %s: %s', console_sock, err) end end) end end -- Start the Instance. local success, data = pcall(dofile, instance_path) if not success then log.error('Failed to run instance: %s', instance_path) os.exit(1) end return 0 end start_instance()
I think that tt start
can provide the following algorithm:
os.dup2
to move real stdin to fileno=5 (for example)|tarantool
and send the lua to its stdinos.dup2
(in the lua) again from fileno=5 to fileno=0console.eval
) should be used only in cases:tt connect -f FILE_NAME
COMMAND | tt connect -f -
-x
option enables a mode where all executed commands are printed to the terminal (trace mode like in bash)--language
option (Lua(\set language lua
), SQL(\set language sql
))Let's say we have a command: tt help crud import
If we run this command without arguments, then tt will fall:
$ tt help crud import
2022/06/21 15:33:57 Whoops! It looks like something is wrong with this version of Tarantool CLI.
Error: Unhandled internal error: runtime error: invalid memory address or nil pointer dereference
Version: Tarantool CLI version <unknown>, linux/amd64. commit: 1b9e738
Stacktrace:
goroutine 1 [running]:
runtime/debug.Stack()
/usr/lib/go/src/runtime/debug/stack.go:24 +0x65
github.com/tarantool/tt/cli/util.InternalError({0x77f67d, 0x1c}, 0xc000153a68?, {0xc000153a20, 0x1, 0x1})
cli/util/util.go:88 +0xa5
main.main.func1()
cli/main.go:18 +0x5f
panic({0x726ce0, 0xa7fd20})
/usr/lib/go/src/runtime/panic.go:838 +0x207
github.com/tarantool/tt/cli/cmd.getInternalHelpFunc.func1(0x72a1e0?, {0xc00007df50?, 0x7712eb?, 0x4?})
cli/cmd/help.go:64 +0x72
github.com/tarantool/tt/cli/modules.RunCmd(0xa8f6a0, {0x7712eb, 0x4}, 0x894e70?, 0xc00000e438, {0xc0000120e0, 0x2, 0x2})
cli/modules/run.go:35 +0x90
github.com/tarantool/tt/cli/cmd.configureHelpCommand.func1(0xf?, {0xabee00?, 0x0, 0xc000122800?})
cli/cmd/help.go:34 +0x1a5
github.com/spf13/cobra.(*Command).Help(0x5ab370?)
The problem is incorrect registration of the module in the modulesInfo
array on long commands.
Temporary workaround:
diff --git a/cli/cmd/help.go b/cli/cmd/help.go
index 49a7975..19b6c64 100644
--- a/cli/cmd/help.go
+++ b/cli/cmd/help.go
@@ -61,6 +61,11 @@ func getInternalHelpFunc(cmd *cobra.Command, help DefaultHelpFunc) modules.Inter
// is enough to check cmd.Name() == "tt".
// - `tt help -I`
// - `tt --help`, `tt -h` or `tt` (look code above).
+
+ case module == nil:
+ fallthrough
case cmd.Name() == "tt", module.IsInternal:
help(cmd, nil)
Helpers need to be modified to support nested commands.
The module will be used to create an application from a template.
When a new instance of tarantool is started in instance.go
, by default the log will be written with block buffering. The stdbuf
utility was used to change this behavior, but it is not installed on macOS by default. Needs to investigate the problem and implement a universal solution.
// Start starts the Instance with the specified parameters.
func (inst *Instance) Start() error {
// By default (when using "glibc") "stdout" is line buffered when connected
// to a TTY and block buffered (one page 4KB) when connected to a pipe / file.
// This is not how we want to log work, so set "stdout" to line buffered mode
// by using "stdbuf" utility. "strderr" is set to no-buffering by default.
//
// Several useful links:
// https://www.pixelbeat.org/programming/stdio_buffering/
// https://man7.org/linux/man-pages/man3/setbuf.3.html
// https://github.com/coreutils/coreutils/blob/master/src/stdbuf.c
inst.Cmd = exec.Command("stdbuf", "-o", "L",
inst.tarantoolPath, "-e", instanceLauncher)
inst.Cmd.Stdout = inst.logger.Writer()
inst.Cmd.Stderr = inst.logger.Writer()
inst.Cmd.Env = append(os.Environ(), "TT_CLI_INSTANCE="+inst.appPath)
inst.Cmd.Env = append(inst.Cmd.Env,
"TT_CLI_CONSOLE_SOCKET="+inst.consoleSocket)
// Imitate the "tarantoolctl".
inst.Cmd.Env = append(inst.Cmd.Env, "TARANTOOLCTL=true")
// Set the sign that the program is running under "tt".
inst.Cmd.Env = append(inst.Cmd.Env, "TT_CLI=true")
// Start an Instance.
if err := inst.Cmd.Start(); err != nil {
return err
}
inst.done = false
return nil
}
Note that the original problem may have been misunderstood by the author.
Now tt completion
is implemented for internal modules. The rocks
module case is where a certain external module is plugged in at compile time. It is necessary to implement tt completion
for this case as well. The connection mechanism must be generalized, because in the future we will be able to connect more modules in the same way.
Github changed policy of cloning repository, and forces now to specify git+https protocol to fetch sources.
There are plenty of rockspecs published in luarocks.org which has old url format: git://github.com.
Hisham already fixed this behavior in luarocks:3.8.0 and we should take his commit to our fork of luarocks
Steps to reproduce:
$ echo 'test' > /tmp/tt
$ mage unit
Running unit tests...
fatal: No names found, cannot describe anything.
? github.com/tarantool/tt/cli [no test files]
? github.com/tarantool/tt/cli/cmd [no test files]
? github.com/tarantool/tt/cli/cmdcontext [no test files]
? github.com/tarantool/tt/cli/codegen [no test files]
--- FAIL: TestConfigureCli (0.00s)
configure_test.go:50:
Error Trace: configure_test.go:50
Error: Expected nil, but got: 0x8
Test: TestConfigureCli
configure_test.go:72:
Error Trace: configure_test.go:72
Error: Expected nil, but got: 0x8
Test: TestConfigureCli
FAIL
FAIL github.com/tarantool/tt/cli/configure 0.002s
? github.com/tarantool/tt/cli/modules [no test files]
? github.com/tarantool/tt/cli/rocks [no test files]
ok github.com/tarantool/tt/cli/running (cached)
ok github.com/tarantool/tt/cli/ttlog (cached)
? github.com/tarantool/tt/cli/util [no test files]
? github.com/tarantool/tt/cli/version [no test files]
FAIL
Error: running "go test ./cli/..." failed with exit code 1
Add the basic option (--yaml
) for tvisor.
Output format for each module:
tt:
status: ok # or error
http: 200 # optional. default 200 for 0, 500 for not 0 exit code
# the other results
example:
tt version --yaml
tt:
status: ok
version: 2.8.1
PS: Also the option can be set by ENV variable TVISORD=yaml[,theotheroptions]
if split(/\s*,\s*/, $ENV{TVISORD})
contains yaml
, then the option have to be set on.
Installs tarantool or tt (or tt modules or something else) into the current bundle.
tt install tarantool=2.6.1
tt remove tarantool
tt install -s tarantool=2.5.1
Options:
-s
- build and installA declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.