Code Monkey home page Code Monkey logo

tidbit's Introduction

Tidbit v3.0 (bleeding edge) -- Tidbit v2.0 (stable)

Master Develop
Build status Build status
Coverage Status Coverage Status

Tidbit is random data generator for Sugar versions 7.8 and later. By optimizing the communications with the database, large amounts of data can be inserted into the system for testing without manual intervention.

Documentation in the wiki!

Please visit the wiki for detailed documentation on using and configuring Tidbit.

Requirements

PHP 5.6+ Sugar Already installed (7.8+ versions)

Installation

To install Tidbit, unpack the Tidbit-v###.tar.bz2 file, and place the Tidbit/ directory within your SugarCRM installation (Tidbit Directory need to be created inside SugarCRM Installation folder).

Download composer

curl -sS https://getcomposer.org/installer | php

Install composer dependencies inside Tidbit directory

./composer.phar install

The only other requirement of Tidbit is that you have an installed and properly configured copy of Sugar in the directory above it.

Installation of Vagrant Stack (Example):

  1. SSH into vagrant stack you are using via command line.

    $ vagrant ssh
    
  2. Navigate to Sugar directory.

    $ cd /
    $ cd var/www/html/ (sugar instance).
    
  3. Download zip file from repo (master.zip) into the sugar instance. e.g. /SugarEnt-Full-7.6.0.0/.

    $ wget (url to zip file)
    
  4. Unzip file (master.zip). Directory created by zip file is called Tidbit-master.

    $ unzip master.zip
    
  5. Change Directory to /Tidbit e.g., mv /Tidbit-master /Tidbit.

    $ mv /Tidbit-master /Tidbit
    
  6. Navigate to /Tidbit and follow instructions under usage, below, from within /Tidbit directory.

    $ cd /Tidbit
    
  7. Install Composer dependencies

    $ ./composer.phar install
    

Configuration

Tidbit has default config files in tidbit_root/config/ folder.
Here:
    - config.php          -- main config file
    - data/*.php          -- customization of filling for separate fields if
                             bean tables
    - relationships/*.php -- customization of filling for separate fields if
                             bean relation tables
Tidbit settings can be fully overrided in tidbit_root/custom/config.php file
by php arrays in the same manner as in original configs

Usage

NOTES:

  • Usage of Tidbit could affect your data in DB Please make sure you have a backup, before running data Generation commands

  • In case of generation csv (--storage csv) We suppose what csv-dump will be used on empty DB, so for speed up, we'll generate values (integer starting with 1) for autoincrement-type fields.

Tidbit uses a command line interface. To run it from the Tidbit directory:

$ ./bin/tidbit (or ./vendor/bin/tidbit for package dependency installation)

Various options are available to control the number of entries generated. To view them:

$ ./bin/tidbit -h

Example usages:

* Clean out existing seed data when generating new data set:
  $ ./bin/tidbit -c

* Insert 500 users:
  $ ./bin/tidbit -u 500
  
* Generate data into csv (mysql is default):
  $ ./bin/tidbit --storage csv

* Generate records for all out-of-box and custom modules, plus find all relationships
  $ ./bin/tidbit --allmodules --allrelationships

* Obliterate all data when generating new records with 400 users:
  $ ./bin/tidbit -o -u 400

* Use profile (pre-existing one: simple, simple_rev2, average, high) file to generate data
  $ ./bin/tidbit -o --profile simple --sugar_path /some/sugar/path

* Use custom profile (located in /path/to/profile/file)
  $ ./bin/tidbit -o --profile /path/to/profile --sugar_path /some/sugar/path

* Generate TeamBasedACL action restrictions for chosen level (check level options in config files)
  $ ./bin/tidbit -o --tba -tba_level full
  
* Controlling INSERT_BATCH_SIZE (MySQL Support only for now)
  $ ./bin/tidbit -o --insert_batch_size 1000

* Setting path to SugarCRM installation
  $ ./bin/tidbit -o --sugar_path /some/sugar/path

* Using DB2 storage example (mysql/oracle/db2 can be used, depending on Sugar installation and DB usage)
  $ ./bin/tidbit -o --sugar_path /some/sugar/path --storage db2

Contributing:

See CONTRIBUTING for how you can contribute changes back into this project.

All Pull Requests should be targeted to "develop" branch and follow PSR2 Code Style Standard To run quick code check use

$ ./composer.phar check-style

or call PHP CS directly

$ ./vendor/bin/phpcs --standard=./ruleset.xml

to run PHPUnit tests locally please use

$ ./composer.phar tests

or call PHPUnit directly

$ ./vendor/bin/phpunit -c ./phpunit.xml.dist

There are automated PR checks enabled on TravisCI (https://travis-ci.org/sugarcrm/Tidbit) For each PR code-style and phpunit tests will be executed for verification

tidbit's People

Contributors

40min avatar asorokin-sugar avatar asuharev avatar chicks avatar dcluniecrm avatar dmulitsa avatar dstiblo-sugarcrm avatar emitya avatar ezhdan-sugarcrm avatar jbarlowsugar avatar mcarroll83 avatar mmarum-sugarcrm avatar ramanenka avatar rsennewald avatar rzhou avatar skingry avatar sstarikov avatar xye avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tidbit's Issues

vendor/autoloader.php in bootstrap.php

Got error when use tidbit,

Fatal error: require_once(): Failed opening required '/usr/local/apache2/htdocs/7.7.1.0/Tidbit-master/vendor/autoload.php' (include_path='.:/usr/local/lib/php') in /usr/local/apache2/htdocs/7.7.1.0/Tidbit-master/bootstrap.php on line 46

—UPDATE: I follow the WIKI Tutorial and it seems it is not updated. Now after I follow the README to install the composer, that error is gone now.

EmailsText not created for Emails

Just tested with default settings using

php install_cli.php -l 1000

which generated 16000 rows on emails table (as expected) but no rows created on emails_text table. This relationship is essentially atomic (neither "module" record makes sense without the other), and may not be easily definable under "tidbit_relationships", since the emails_text table doesn't have it's own true unique ID key, but instead uses the email_id column as the primary key, which is also the foreign key for the related email record. So the entry might look similar to:

$tidbit_relationships['EmailText'] = array(
    'Emails' => array(
        'self' => 'email_id',
        'you' => 'email_id',
        'table' => 'emails_text'
    ),
);

But it's not at all clear (to me) if the above would be the way to go. This may be a documentation issue (in terms of making it clear how to handle various relationships), or may be a needed addition to the application (basically defining how this sort of relationship would get handled), or may reflect a need in the application for handling "tight" or "non-optional" one-to-one relationships, or even a need to hard-code this specific use-case into the application (as it seems like it is very atypical and likely not the "right" way we would implement such a pattern if we started from scratch).

Conceptually, I think this relationship would be considered "bijective" (if that is at all helpful to the power-math-nerds out there):

https://en.wikipedia.org/wiki/Bijection

On DB2 vagrant stack not able to create account records.

[vagrant@php54-db2-stack Tidbit]$ php install_cli.php -l 10
Constructing
0 Teams
0 EmailAddresses
0 ACLRoles
0 Users
10 Accounts
0 Quotes
0 ProductBundles
0 Products
0 Calls
0 Emails
0 Contacts
0 Leads
0 Opportunities
0 Cases
0 Bugs
0 Meetings
0 Tasks
0 Notes
0 Documents
0 Categories
0 KBContents

With Clean Mode OFF
With Turbo Mode OFF
With Transaction Batch Mode 0
With Obliterate Mode OFF
With Existing Users Mode OFF
With ActivityStream Populating Mode OFF
With 10 teams in Team Sets
With Team-based ACL Mode OFF
With Team-based Restriction Level OFF

Processing Module Teams
Inserting 0 records.

    *Hitting DB... Database failure.* Please refer to sugarcrm.log for details.Database failure. Please refer to sugarcrm.log for details.[vagrant@php54-db2-stack Tidbit]$

Sugarcrm.log says following error:

Mon Jun 20 23:44:45 2016 [7150][-none-][FATAL] INSERT QUERY FAILED Query Failed: INSERT INTO team_sets ( id, name, team_md5, team_count, date_modified) VALUES ('a3c5407a-377b-11e6-99dc-56847afe9799', '55ff757c5f6e588a541befae3e662c68', '55ff757c5f6e588a541befae3e662c68', 10, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3c68246-377b-11e6-9d0a-56847afe9799', 'b862cc37cba09f330899c1cafe4b344c', 'b862cc37cba09f330899c1cafe4b344c', 9, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3c71c2e-377b-11e6-ba05-56847afe9799', '93960e5e94d61b97790dac88c75bca20', '93960e5e94d61b97790dac88c75bca20', 8, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3c9a34a-377b-11e6-8e7d-56847afe9799', 'f033b79788d955355b172a3dce6f0140', 'f033b79788d955355b172a3dce6f0140', 7, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3ca10e6-377b-11e6-aaf4-56847afe9799', '2981efc8e379096e8a27858ab337fc36', '2981efc8e379096e8a27858ab337fc36', 6, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3cc90e6-377b-11e6-bf72-56847afe9799', '156b61adf6525511793e4b30dbd1993e', '156b61adf6525511793e4b30dbd1993e', 5, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3ccf662-377b-11e6-902f-56847afe9799', '98ff04595d28116dd8787b468b612de0', '98ff04595d28116dd8787b468b612de0', 4, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3cd50d0-377b-11e6-8bb2-56847afe9799', '9d8cae4502ff595ab9020ccc6ab41d32', '9d8cae4502ff595ab9020ccc6ab41d32', 3, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3cd9dba-377b-11e6-8ab2-56847afe9799', '9bb1e38db063660e9f22ba67d8688f2b', '9bb1e38db063660e9f22ba67d8688f2b', 2, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('1 ', '69bc0547e5860dca6da3f25368522dc9', '69bc0547e5860dca6da3f25368522dc9', 1, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3cdf116-377b-11e6-a4ff-56847afe9799', '3b471e58a0c491294f43643eb307ff53', '3b471e58a0c491294f43643eb307ff53', 10, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3d0b7ca-377b-11e6-a7ac-56847afe9799', 'ec230164930cf8cc5e3a95387c488a6a', 'ec230164930cf8cc5e3a95387c488a6a', 9, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3d145be-377b-11e6-9e78-56847afe9799', '910347f19b73d6c9db8548ca147907ef', '910347f19b73d6c9db8548ca147907ef', 8, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3d3a048-377b-11e6-84fa-56847afe9799', '360cf04ba11cfc68f5a67b23d0976e48', '360cf04ba11cfc68f5a67b23d0976e48', 7, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3d41ff0-377b-11e6-b565-56847afe9799', '4c869f72f3709197684d4dda2d3106d6', '4c869f72f3709197684d4dda2d3106d6', 6, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3d4b4ec-377b-11e6-b8cd-56847afe9799', '8b1c29b3bc73b5a7ff1b26d5eff24bfc', '8b1c29b3bc73b5a7ff1b26d5eff24bfc', 5, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3d6db14-377b-11e6-9d2b-56847afe9799', 'a46b2a94f6e30a5a21aae3b14fc77427', 'a46b2a94f6e30a5a21aae3b14fc77427', 4, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3d7413a-377b-11e6-ab26-56847afe9799', '3b1e226bda2750fc906a708090a89c9e', '3b1e226bda2750fc906a708090a89c9e', 3, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('a3d794dc-377b-11e6-a6bc-56847afe9799', '41d3cc50703336e99def1350b04c7a2c', '41d3cc50703336e99def1350b04c7a2c', 2, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS')),('fb87d846-33b5-11e6-bab7-56847afe9799', '088920ca99d7039dcbbb989aa7fc33ac', '088920ca99d7039dcbbb989aa7fc33ac', 1, to_date('2016-06-21 06:44:43', 'YYYY-MM-DD HH24:MI:SS'));: IBM_DB2 statement error 23505: [IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "1" constrains table "DB2INST1.TEAM_SETS" from having duplicate values for the index key. SQLSTATE=23505 SQLCODE=-803

Activity Streams and Teams issues

In the past few days I've experienced a couple of different issues regarding activity streams creation.

On MySQL, the Activity Streams records were created, but not visible. In the sense that an admin user could not see them, but in the database records were created.

Sample command: ./Tidbit-2.0.0/bin/tidbit -o -d -t --profile simple --as_populate --sugar_path /my/path/to/820ent/

Sample output:
Constructing
4 Teams
20 Users
12000 EmailAddresses
10 ACLRoles
1000 Accounts
24000 Calls
4000 Contacts
4000 Leads
2000 Opportunities
4000 Cases
8000 Meetings
4000 Tasks
4000 Notes

With Clean Mode OFF
With Turbo Mode ON
With Transaction Batch Mode 0
With Obliterate Mode ON
With Existing Users Mode OFF
With ActivityStream Populating Mode ON
With 10 teams in Team Sets
With Team-based ACL Mode OFF
With Team-based Restriction Level OFF

With the same command, the system had problems with Teams as well (users not part of the global team as an example). Even after repairing teams, Activity Streams records were not visible.

On DB2, the Activity Stream records were not even getting generated on the database.
Sample command: ./bin/tidbit -e -x 2 -t -c -o --profile simple --as_populate --storage db2 --sugar_path /var/www/html/SugarEnt-8.0.0/

Can someone please check out where the problems might be?

Thanks!

Feature: Remove data added by Tidbit

It would be great if there was a command to remove the data created by Tidbit. An example where this would be helpful is if a User mis-types the factor or needs to add more data later. Currently, Tidbit will attempt to create exactly the same data again, leading to database failures for duplicate entries.

Doesn't support CE

Hi,

Been fiddling around getting this up and running today - but because of the requirement for TEAMS - this doesn't run against the community edition, no?

Users custom table is not cleared with -c or -o

Steps

  1. Create a custom field in Users
  2. Populate data with Tidbit (eg: ./bin/tidbit --profile simple --allmodules --allrelationships)
  3. Re-populate data with Tidbit using either -c or -o option

Expected results
Tidbit completes inserts of data correctly

Actual result
Script will eventually fail

Current workaround is to truncate the users_cstm table before re-running Tidbit

Create Classic/Basic generator for common modules

Create Classic/Basic generator for common modules

We want to have Tidbit operate in a similar way (interface) and Basic Generators should not different from Custom Generators on Interface levels

Multienum Fields Data

Are Tidbit ready for multienum fields?

I have generated data for a custom module with a multienum field and left all values blank or with default value (^defaultvalue^)

Best Regards,
Rubén Recacha.

clean mode should not delete the team sets that have been used by other existing records

clean mode will only delete the seed data generated by tidbit.

reproduce steps:

  1. create teams "team1", "team2"
  2. add new account record "account test 1" with team sets "team1", "team2", "admin"
  3. run following command ./bin/tidbit -c --profile simple --sugar_path /Users/xye/htdocs/mysql/ent7920
    (you may change the config/profiles/simple.php file to only add one record for each of the modules)
  4. login to admin, go to account record "account test 1", observe on the teams field.

this field is empty, now ONLY admin user can see this record "account test 1", no one from team1 or team 2 can see this record any more.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.