CORTX QUICK START GUIDE.

the current workflow is :

1.Set up your build and test environment
2. Getting cortx-motr ready
3. Getting Cortx-S3 ready
4. A video from CORTX engineer Kevin Price demonstrating the above three steps
5. Contributing
6. Contributing to Cortx Motr
7. Contributing to Cortx S3
.....

I would like to suggest the following:

1. Set up your build and test environment
2. Prerequisites (merge general information from Getting Cortx-motr ready and Getting Cortx-S3 ready)
3. Getting Cortx-motr ready
4. Getting Cortx-S3 ready
5. A video from CORTX engineer Kevin Price demonstrating the above three steps
6. Contributing
7. Code commits and reviews (merge Contributing to Cortx Motr and Contributing to Cortx S3)

As you can see the instructions in Getting cortx-motr ready and Getting Cortx-S3 ready has some overlaps, we can move the general items, such git, python, kernel, pip version check, update, and installation, etc to prerequisites section, also provide debug guidance. We keep anything specific to the cortx-motr and cortx-s3 guidance to the corresponding sections.

Also, Contributing to Cortx Motr and Contributing to Cortx S3 provides very general information, which overlaps almost completely. Merge those two into one is good enough.

I think we should standardize git protocols among all submodules

Most if not all submodules protocol uses https (e.g. https://github.com/Seagate/cortx-s3server.git).
However, in the cortxs3 documentation, it recommends (instruct) to setup ssh access which won't work when you git clone recursively. You still need to setup PAT (as noted in doc) and I think we should decide which protocol to stick with.

I personally prefer ssh but I guess whichever should do just fine as long as users do not need to setup double access.

https://github.com/Seagate/cortx/blob/master/doc/CortxMotrQuickStart.md contains a link to an internal Seagate site, which would not be available to a community member. Please move the resource to a public location and update the link. Thanks.

Reference:
#$ scp -p -P 29418 g${GID}@gerrit.mero.colo.seagate.com:hooks/commit-msg ${gitdir}/hooks/commit-msg

Few times a day, when trying to access github.com/Seagate, I am getting

Following the link gets me to

which after some time results in

The only known way to fix this is by following a little dance of clearing all browser cookies for ssologin.seagate.com.

I have Seagate VPN active, all other services work fine.

Update readme file to address new contributors.

[root@ssc-vm-0925 cortx-motr]# ./scripts/m0 run-st 01net
----- run_st 01net -----
<< 01net >>
insmod: ERROR: could not insert module /home/670951/cortx-motr/extra-libs/galois/src/linux_kernel/galois.ko: File exists
Error unloading /home/670951/cortx-motr/m0tr.ko.
0.00user 0.02system 0:00.04elapsed 85%CPU (0avgtext+0avgdata 4728maxresident)k
24inputs+8outputs (0major+5948minor)pagefaults 0swaps

I followed the https://github.com/Seagate/cortx/blob/master/doc/Build-Installation-FAQ.md Qs3

couldn't see which module is using m0tr

[root@ssc-vm-0925 cortx-motr]# lsmod
Module Size Used by
m0tr 13920279 5
galois 22944 1 m0tr
loop 28072 20
ksocklnd 183608 1
lnet 586401 3 m0tr,ksocklnd
libcfs 415252 2 lnet,ksocklnd
rpcsec_gss_krb5 35549 0
auth_rpcgss 59415 1 rpcsec_gss_krb5
nfsv4 583113 1
dns_resolver 13140 1 nfsv4
nfs 261876 2 nfsv4
lockd 98048 1 nfs
grace 13515 1 lockd
fscache 64984 2 nfs,nfsv4
ipmi_devintf 17459 0
ipmi_msghandler 56728 1 ipmi_devintf
sunrpc 354099 9 nfs,rpcsec_gss_krb5,auth_rpcgss,lockd,nfsv4
nfit 55479 0
ppdev 17671 0
libnvdimm 155545 1 nfit
iosf_mbi 15582 0
crc32_pclmul 13133 0
ghash_clmulni_intel 13273 0
aesni_intel 189456 0
lrw 13286 1 aesni_intel
gf128mul 15139 1 lrw
glue_helper 13990 1 aesni_intel
ablk_helper 13597 1 aesni_intel
cryptd 21190 3 ghash_clmulni_intel,aesni_intel,ablk_helper
joydev 17389 0
pcspkr 12718 0
virtio_rng 13029 0
sg 40721 0
virtio_balloon 18015 0
i2c_piix4 22401 0
parport_pc 28205 0
parport 46395 2 ppdev,parport_pc
ip_tables 27126 0
ext4 584153 6
mbcache 14958 1 ext4
jbd2 107478 1 ext4
sd_mod 46281 3
crc_t10dif 12912 1 sd_mod
crct10dif_generic 12647 0
sr_mod 22416 0
cdrom 42556 1 sr_mod
virtio_console 28076 1
virtio_scsi 18463 2
virtio_net 28063 0
virtio_blk 18323 4
ata_generic 12923 0
pata_acpi 13053 0
qxl 59032 1
drm_kms_helper 186531 1 qxl
syscopyarea 12529 1 drm_kms_helper
sysfillrect 12701 1 drm_kms_helper
sysimgblt 12640 1 drm_kms_helper
fb_sys_fops 12703 1 drm_kms_helper
ttm 96673 1 qxl
ata_piix 35052 0
crct10dif_pclmul 14307 1
drm 456166 4 qxl,ttm,drm_kms_helper
crct10dif_common 12595 3 crct10dif_pclmul,crct10dif_generic,crc_t10dif
crc32c_intel 22094 0
libata 243133 3 pata_acpi,ata_generic,ata_piix
serio_raw 13434 0
virtio_pci 22985 0
floppy 69432 0
virtio_ring 22746 7 virtio_blk,virtio_net,virtio_pci,virtio_rng,virtio_balloon,virtio_console,virtio_scsi
virtio 14959 7 virtio_blk,virtio_net,virtio_pci,virtio_rng,virtio_balloon,virtio_console,virtio_scsi
drm_panel_orientation_quirks 17180 1 drm
dm_mirror 22289 0
dm_region_hash 20813 1 dm_mirror
dm_log 18411 2 dm_region_hash,dm_mirror
dm_mod 124501 25 dm_log,dm_mirror

[root@ssc-vm-0925 cortx-motr]# rmmod m0ctl
rmmod: ERROR: Module m0ctl is not currently loaded
[root@ssc-vm-0925 cortx-motr]# rmmod m0tr
rmmod: ERROR: Module m0tr is in use
[root@ssc-vm-0925 cortx-motr]# rmmod galois
rmmod: ERROR: Module galois is in use by: m0tr

I rebooted my VM, then no error.

Hello-

I installed the Centos 7.7 ISO http://mirrors.oit.uci.edu/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso specified by https://github.com/Seagate/cortx/blob/master/doc/VIRTUAL_MACHINE.md. The VM setup document requires kernel version 3.10.0-1062.12.1 but I ended up with kernel version 3.10.0-1127.13.1.el7.x86_64. I did not deliberately upgrade the kernel, but I did update the distro prior to use (this is normal, I think, to capture any security updates or bug fixes). Is this kernel version OK to use? If so, can we update the documentation to indicate what kernel versions are OK, if there is a dependency?

Thanks,
-Tim

https://github.com/Seagate/cortx/blob/master/doc/CortxMotrQuickStart.md has us run sudo ./scripts/install-build-deps to install the build dependencies. The script fails when fetching the kernel source:

TASK [base-os : install kernel-devel for the current kernel] ***************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "No package matching 'kernel-devel-3.10.0-1062.el7.x86_64' found available, installed or updated", "rc": 126, "results": ["No package matching 'kernel-devel-3.10.0-1062.el7.x86_64' found available, installed or updated"]}
...ignoring
msg: No package matching 'kernel-devel-3.10.0-1062.el7.x86_64' found available, installed or updated
results: No package matching 'kernel-devel-3.10.0-1062.el7.x86_64' found available, installed or updated

However, if the OS release is updated (sudo yum update) to pick up the current versions of Centos 7.7 packages, we get kernel 3.10.0-1127.13.1.el7.x86_64 which does have the devel package available. After an update ./scripts/install-build-deps runs with no errors.

CortX team told me 3.10.0-1062 is required, however the build process won't complete unless the system is upgraded to 3.10.0-1127. Please either change the kernel requirement away from 3.10.0-1062 or make the 3.10.0-1062 dev source available on Github.

In general we need to be able to update the Centos 7.7 distribution prior to use. It is important to keep these systems patched and updated for security and bug fixes. The update process brings an updated kernel. We should support current kernels, or understand the requirement that drives the need for an old kernel.

Thanks.

[root@ssc-vm-0925 cortx-motr]# make doc
make: doxygen: Command not found
make: *** [doc-doxygen] Error 127

https://github.com/Seagate/cortx-motr/tree/dev#go-deep

Please fill the placeholders in the VA document (installation guide).

In the parent repo the "sub-modules" are not pointing to the latest changes in the child repo.

We need a way to automatically update these or have a process where they are updated.

Can somebody please explain how can I get rid of this noisy message after each my push to GitHub?

Text:

Thanks for your contribution!
The CLA bot has flagged your contribution as not having a Contributor License Agreement
in place. Note that this is not needed in the overwhelming majority of instances and this warning will usually be ignored.
The code reviewers will make a determination and may ask you to sign a CLA or may choose to ignore this warning.
More information about this can be found here.

The proposed hyperlink leads to a short explanation what CLA means but doesn't explain what as a valid developer should I do next.
Just to clarify:

1. I keep committing from Citrix only, so my git uses only my seagate email address.
2. I use a dedicated GH account where Seagate email is set as primary.

Thanks in advance for any help.

cc @johnbent, @mukul-seagate11, @venkuppu-chn, @xahmad, @gauravchaudhari02

After going open source, in the GitHub and Codacy.rst document, the following changes must be made.

• PAT related content must be removed

• SSO related content must be removed

• Establish links between documents

#107

[root@ssc-vm-0922 670951]# hctl bootstrap --mkfs singlenode.yaml
2020-08-15 01:06:36: Generating cluster configuration.../opt/seagate/cortx/hare/libexec/hare-bootstrap: line 234: m0confgen: command not found

I found that m0confgen is located at the following locations:

cortx-motr/st/m0confgen
cortx-motr/utils/m0confgen

cortx-s3server/third_party/mero/st/m0confgen
cortx-s3server/third_party/mero/utils/m0confgen

not in this location:

cortx-hare/utils/

which directory should we run this command?

https://github.com/Seagate/cortx/blob/master/doc/Cluster_Setup.md#15-bootstrap-the-cluster

From my understanding, we changed mero to motr
but in the S3 (git clone --recursive [email protected]:Seagate/cortx-s3server.git -b main)
still see mero

Submodule path 'third_party/mero': checked out 'a03ba35788f5650678e1086ce8c1edf0644f20c1'
Submodule path 'third_party/mero/extra-libs/galois': checked out '0975162a5fa8d0de3758756fc006097ef24a

where can we find the search box?

https://github.com/Seagate/cortx/blob/master/doc/CortxS3ServerQuickStart.md#testing-specific-motr-version-with-cortx-s3server

Commands:

**run-all**              Run unit and system tests.

**dist-check**           Create and validate Motr distribution archive and rpm.

**check-everything**  =  (rebuild + dist-check + run-all) * 2
                     *NOTE* Please run this command before landing.
                     .
                     The command is executed twice (note '* 2' above):
                     the 2nd time with './configure --disable-m0-asserts'.

**clean [-f]**           Execute 'make distclean' in Motr sources directory.
                     '-f' will also remove untracked files and directories
                     (i.e. those not added to the git repository).
                     *CAUTION* Be careful with '-f' option!

location: cortx-motr/scripts/m0

https://github.com/Seagate/cortx/blob/master/doc/CortxMotrQuickStart.md

@gauravchaudhari02 , can you please add the submodule pointers in this repo?

run ./scripts/m0 run-ut -T in the cortx-motr directory
nothing comes out, it's just hanging there with

----- run_ut -T -----

A significant portion of English-speaking world has Write access to Cortx repositories.

E.g., https://github.com/Seagate/cortx-motr/settings/access:

⚠️ This situation is dangerous.

Write access should be restricted to those who actively push to the project.

All Cortx users and outside collaborators should be given Read access to all Cortx repositories by default. This level of access is more than adequate for posting patches.

Read more about permission levels for repositories owned by an organization.

BUILD_ENVIRONMENT.md says, Currently CORTX requires the CentOS 7.7.1908 distribution running the 3.10.0-1062.12.1 kernel. I feel we are supporting 3.10.0-1062.el7.x86_64 Kernel version as of now. Hence document need to be updated.

Current mero repository works with Centos7.7.

Plan is to add support for Centos8 (starting with CentOS8.2)

@huanghua78 @justinzw @johnbent

Hello-

After the S3 build, the jenkins unit tests fail for missing s3amicli. The error message states:

Check S3CMD...OK
s3iamcli not installed
Please install s3iamcli using rpm built from /rpms/s3iamcli/

---

*** ABORTED ***

---

However the s3iamcli rpm is not present anywhere in the S3 tree, although I do see the build script in rpms/s3iamcli. Is there a missing compile step?

Thanks.

The other error and warning messages are in here: s3test.txt

https://www.zdnet.com/article/github-to-replace-master-with-alternative-term-to-avoid-slavery-references/

doc/hld/hld-template.rst.in in dev/architecture branch

https://github.com/Seagate/cortx/blob/master/doc/SuggestedDocumentConversions.md#motr-high-level-designs

I am testing CORTX using 60000 MNIST images.
mnist_up.zip is my python program to upload those images to CORTX. It worked fine with loading a portion of those images (eg. 10000), but caused “gateway timeout” error if I tried to upload more than 30000 images.

I also tried to use “aws s3 cp” command (s3Copy.zip) to upload those images one by one and it caused the same “gateway error” starting from img27284. After this error, all aws commands not work anymore (504 error).

Submitting contributions link to demo video

404 NOT FOUND

1. https://github.com/Seagate/cortx/blob/master/doc/be/EOSS3IAMDD.rst#references

2. https://github.com/Seagate/cortx/blob/master/doc/be/CORTX-S3OVERVIEW.rst

search keyword google, you should see those three links

2a https://github.com/Seagate/cortx/blob/master/doc/be/CORTX-S3OVERVIEW.rst#cortx-s3-object-to-cortx-motr-object-mapping

2b https://github.com/Seagate/cortx/blob/master/doc/be/CORTX-S3OVERVIEW.rst#cortx-s3-metadata

2c https://github.com/Seagate/cortx/blob/master/doc/be/CORTX-S3OVERVIEW.rst#cortx-s3-apls-support

Add documentation about how DCO and CLA will be required

• Set up DCO and CLA bots
• DCO will be always required
• Corporate CLA will be required when a contributor belongs to a corporation and is making a large contributions; in typical cases CLA bot can/will be overridden
• When corporate CLA is deemed to be required, we’ll ask them to sign Apache CLA

SSO prompt appears several times a day. It is followed by the Username/Password prompt. This is bad UX.

Solution: increase SSO time-out interval to 12 hours.

@gauravchaudhari02 @venkuppu-chn , some community members are directly merging their own contributions! :)
For example, #25.
Can we make it so that everyone, with just the exception of a few gatekeepers, have only read access to our soon-to-be open source modules?

To be in sync with the other component branching strategy, the default branch shouldn't be master (which is also a contentious term).
Kindly correct this to align this with the component repo strategy.

Problem

“We should subtract 10% from [our team's] capacity for inefficient tools we are using.”
-- Cortx developer

• SharePoint documents are slow 🐌 to open and rather taxing on the CPU. 💻🔥
  • PowerPoint presentations are particularly bad in this respect: multi-slide presentations take ages to load or, in case of slow network, may not open at all.
• UX of SharePoint products is much worse than that of Google Documents, which engineers used before. The shortcomings are too many to list, but how about this one:
  • SharePoint Excel doesn't allow dragging of rows to rearrange them.
• Microsoft Teams' UX is poor.
  • Message editor is unreliable, quirky, and difficult to use.
  • Desktop app requires regular restarts (sometimes just stops updating itself).
  • The app is slow to reload after network disconnection.
  • Some messages are impossible to get a link to.
  • Cmd-C does “Select all” instead of “Copy”. Wha?

This list is by no means complete.

Mitigation

Avoid using tools that cause friction. Do more with GitHub issues, pull requests, and wiki. Take PC3 refresher.

PC3 is meant to provide an optimal collaboration model for commercial software projects. Broadly, PC3 helps an organization build consistently good software, cheaply, and rapidly.

Solution

Ditch Microsoft Office365/SharePoint in favour of NextCloud.

[root@ssc-vm-0922 670951]# cp /opt/seagate/cortx/hare/share/cfgen/examples/singlenode.yaml
cp: missing destination file operand after ‘/opt/seagate/cortx/hare/share/cfgen/examples/singlenode.yaml’
Try 'cp --help' for more information.

which directory should we run this command and where should we put the copy at?

https://github.com/Seagate/cortx/blob/master/doc/Cluster_Setup.md#14-prepare-the-cdf

I downloaded the vm image for virtual box mentioned in directory : https://seagatetechnology-my.sharepoint.com/:f:/g/personal/basavaraj_kirunge_seagate_com/EgkMhqmUbIdFs8Tr9N-bHecBtMDwm02QbG0T8vS7TLOdUg?e=aiRHBr. However, while trying to install dependent packages like lustre using

sudo ./scripts/install-build-deps

it gives the following error

This directory is part of the 'Local VM setup' https://github.com/Seagate/cortx/blob/master/doc/LocalVMSetup.md#1-download-the-vm

BE components image is not clear, hard to read, see here :

https://github.com/Seagate/cortx/blob/master/doc/be/BE_TheMetadataBackend.md#be-components-relationship

https://github.com/Seagate/cortx/blob/master/doc/be/images/be-components.png

There is a failure while setting up dev environment for cortx-s3server.

I saw this failure in the script/env/dev/init.sh script that is run while trying to setup a development environment for cortx-s3server.
The failure happens while installing the packages bundled in ansible/awscli-bundle.zip. The install script in the bundle fails suggesting duplicates for some of the packages. I was able to circumvent the problem by deleting the duplicates from the package folder and then running the install script manually. After that, I commented out the relevant portion from the ansible/common_dev_setup.yml and ran the init.sh script again. This time it ran to completion without any failures.
Here, if the diff between my changed copy and the original file common_dev_setup.yml file.

cortx-s3server]# diff ansible/common_dev_setup.yml ansible/common_dev_setup.orig.yml
57a58,67

• name: Install and configure awscli on RHEL
  when: ansible_distribution == "RedHat"
  command: "{{ item }}"
  with_items:
  • curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
  • unzip awscli-bundle.zip
  • ./awscli-bundle/install -b ~/bin/aws
  • rm -rf awscli*
    no_log: true

@amolkongre , can we have someone update the link in the README for the S3 Opengrok please?

Hello-

S3 rebuild-all script fails:
server/s3_bucket_metadata_v1.h:25:30: fatal error: gtest/gtest_prod.h: No such file or directory
#include <gtest/gtest_prod.h>

See attached complete build output.

Thanks,
-Tim
builderrs.ttw.txt

For some reason, the style of file names in doc/ has been changed. E.g., coding-style.md became CodingStyle.md. Is there any reason for this? If not, please revert to the old style and use it for both existing and new files.

The S3 build process fails with "bazel: command not found". Bazel is not mentioned in any of the build preparation guides and is not available on the standard Centos repos. We need instructions on which version of bazel to use, and how to safely install it. It should also be included in the list of build prerequisites somewhere. Thanks.

[root@ssc-vm-0922 test]# git clone --recursive http://gerrit-sage.dco.colo.seagate.com:8080/mero
Cloning into 'mero'...
fatal: unable to access 'http://gerrit-sage.dco.colo.seagate.com:8080/mero/': Could not resolve host: gerrit-sage.dco.colo.seagate.com; Name or service not known

[root@ssc-vm-0922 test]# git clone --recursive https://github.com/seagate-ssg/mero.git
Cloning into 'mero'...
Username for 'https://github.com':
Password for 'https://[email protected]':
remote: Repository not found.
fatal: repository 'https://github.com/seagate-ssg/mero.git/' not found

Doc: BUILD_ENVIRONMENT.md
Error:

Last login: Sat May 9 22:49:14 2020
LNET configure error 100: Network is down
[root@localhost ~]# ls
anaconda-ks.cfg
[root@localhost ~]# mkdir dk
[root@localhost ~]# cd dk
[root@localhost dk]# git clone ssh://[email protected]:29418/s3server && scp -p -P 29418 [email protected]:hooks/commit-msg s3server/.git/hooks/
Cloning into 's3server'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Create a local repository requires pip, but pip is not installed by any of the preparation scripts or documented.

please feel free to create a new Discussion in our GitHub Team Discussions room:

https://github.com/orgs/Seagate/teams/seagate-cortx-innersource/discussions

404

This command (as in https://github.com/Seagate/cortx/blob/master/doc/CortxMotrQuickStart.md) doesn't work:

git clone --recursive [email protected]:Seagate/cortx-motr.git -b main

Correct command is git clone --recursive [email protected]:Seagate/cortx-motr.git -b dev

either we need to rename "dev" branch as "main" or update specs.

I prefer 'master', as this is typical for repositories in git.

Hey @xahmad , did CLA stop working on this repo? @gauravchaudhari02 do you know anything about this?

Hello,
source-code-browser mentioned at https://github.com/Seagate/cortx#source-code-browser is not working. http://ssc-vm-c-192.colo.seagate.com:8090/source/ gives "This site can’t be reached"

For example, requirement for RoCE and TCP connectivity.
@ipoddubnyy - please add other known restrictions