Comments (23)
I was able to get a hold of GitHub support on the phone and they had a similar report from another Colorado customer and they brought the IP address of their load balancer we were also having issue with out of the loop. The issue seems to be resolved from our end and I can do quick clones now. Looks like the Jenkins jobs are also able to checkout now.
This issue should be resolved so let the Lab team know otherwise.
from cortx-re.
@mukul-seagate11 We face this issue almost daily in Jenkins build process. Can we raise ticket with GitHub for this.
from cortx-re.
@mukul-seagate11 We face this issue almost daily in Jenkins build process. Can we raise ticket with GitHub for this.
Sure, we can take this forward with GitHub call
from cortx-re.
This issue/pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
from cortx-re.
@mukul-seagate11 As updated by Dhruv in today's call, Can you please raise support ticket for this with GitHub team.
from cortx-re.
@mukul-seagate11 As updated by Dhruv in today's call, Can you please raise support ticket for this with GitHub team.
OK sure
from cortx-re.
from cortx-re.
Absorbing random failures in pipeline
http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/NFS/53/
http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/S3server/155/console
from cortx-re.
+1
http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/Hare/160/console
from cortx-re.
+1
http://eos-jenkins.colo.seagate.com/job/Cortx-Dev/job/RHEL-7.7/job/Motr/115//console
This issue occurred between 25 th Aug 11 PM IST to 26 th Aug 2.20 AM IST
from cortx-re.
Update : The checkout is working on the above mentioned time frame, But still getting random failure on checkout
from cortx-re.
Casey Mokricky (GitHub Support)
Aug 25, 2020, 4:16 PM UTC
Hi Mukul,
Thank you for contacting GitHub Enterprise Support.
Just to make sure I'm understanding correctly, it sounds like your git client is having performance issues on a CloudForm VM instance. Is that correct?
If so, please prefix your git command with the GIT_TRACE_PERFORMANCE environment variable and respond with any output. For example:
GIT_TRACE_PERFORMANCE=1 git clone https://github.com/my/repo.git
Feel free to include any additional information as well.
Thank you,
Casey
from cortx-re.
@mukul-seagate11 - We are observing this issue quite frequently now. @gowthamchinna has collected debug logs as below. Git checkout is taking around 14 mins which is causing build failures.
[root@4be3b7e549a9 /]# time GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://<PAT>@github.com/Seagate/cortx-csm.git
02:32:13.129992 git.c:371 trace: built-in: git 'clone' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
Cloning into 'cortx-csm'...
02:32:13.133696 run-command.c:369 trace: run_command: 'git-remote-https' 'origin' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
* Couldn't find host github.com in the .netrc file; using defaults
* About to connect() to github.com port 443 (#0)
* Trying 140.82.114.4...
* Connected to github.com (140.82.114.4) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=github.com,O="GitHub, Inc.",L=San Francisco,ST=California,C=US
* start date: May 05 00:00:00 2020 GMT
* expire date: May 10 12:00:00 2022 GMT
* common name: github.com
* issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache
< HTTP/1.1 401 Authorization Required
< Server: GitHub Babel 2.0
< Content-Type: text/plain
< Content-Length: 21
< WWW-Authenticate: Basic realm="GitHub"
< X-GitHub-Request-Id: 2A98:2BC3:3987949:54AED2F:5F39EC2D
< X-Frame-Options: DENY
<
* Ignoring the response-body
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://<PAT>@github.com/Seagate/cortx-csm.git/info/refs?service=git-upload-pack'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:39879CA:54AED5C:5F39EC2D
< X-Frame-Options: DENY
<
* Connection #0 to host github.com left intact
02:32:15.583453 run-command.c:369 trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.584471 exec_cmd.c:118 trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.586117 git.c:371 trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> POST /Seagate/cortx-csm.git/git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 6214
* upload completely sent off: 6214 out of 6214 bytes
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:3987A2E:54AEE1E:5F39EC2E
< X-Frame-Options: DENY
<
remote: Enumerating objects: 291, done.
remote: Counting objects: 100% (291/291), done.
remote: Compressing objects: 100% (213/213), done.
02:32:16.435083 run-command.c:369 trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.436121 exec_cmd.c:118 trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.437700 git.c:371 trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
* Connection #0 to host github.com left intactiB | 74.00 KiB/s
remote: Total 56013 (delta 132), reused 143 (delta 72), pack-reused 55722
Receiving objects: 100% (56013/56013), 41.69 MiB | 68.00 KiB/s, done.
Resolving deltas: 100% (25949/25949), done.
02:46:30.154135 run-command.c:369 trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.154769 exec_cmd.c:118 trace: exec: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.156481 git.c:371 trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
real 14m17.110s
user 0m3.559s
sys 0m1.760s
from cortx-re.
@mukul-seagate11 @gowthamchinna - Currently all Jenkins jobs are failing at git checkout
stage. I suspect this SSC VM specific issue. I have raised https://jts.seagate.com/browse/LRL-391 for the same.
CC: @amolkongre @priyankdal
from cortx-re.
This appears to be an issue site wide at LCO and not just lab specific. We are starting a Sev1 with IT Networking at Seagate.
from cortx-re.
The issue seems to be with Github load balancers. If we use a different load balancer, we see expected results.
[509589@ssc-vm-0001 ~]$ git clone [email protected]:Seagate/cortx-csm.git
Cloning into 'cortx-csm'...
The authenticity of host 'lb-192-30-255-113-sea.github.com (192.30.255.113)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
RSA key fingerprint is MD5:16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lb-192-30-255-113-sea.github.com,192.30.255.113' (RSA) to the list of known hosts.
remote: Enumerating objects: 372, done.
remote: Counting objects: 100% (372/372), done.
remote: Compressing objects: 100% (285/285), done.
remote: Total 57547 (delta 160), reused 146 (delta 77), pack-reused 57175
Receiving objects: 100% (57547/57547), 42.16 MiB | 20.98 MiB/s, done.
Resolving deltas: 100% (27082/27082), done.
[509589@ssc-vm-0001 ~]$ git clone [email protected]:Seagate/cortx-csm.git
Cloning into 'cortx-csm'...
remote: Enumerating objects: 372, done.
remote: Counting objects: 100% (372/372), done.
remote: Compressing objects: 100% (285/285), done.
Receiving objects: 17% (9783/57547), 3.25 MiB | 38.00 KiB/s
from cortx-re.
I've submitted a ticket here with GitHub support...
https://enterprise.githubsupport.com/hc/en-us/requests/814338
I've also tried to call them and haven't been able to reach anyone.
I am still working with IT as they could possibly route traffic a different direction if needed. However this is more risky as it affects all of the LCO site.
from cortx-re.
Issue resolved
from cortx-re.
@mukul-seagate11 Issue happening again
from cortx-re.
This issue/pull request has been automatically marked as stale because it has not had recent activity. Tagging @shailesh-vaidya
from cortx-re.
@gowthamchinna I feel we are not observing this issue again. Could you please confirm and if yes mark this one as closed.
from cortx-re.
This issue/pull request has been automatically marked as stale because it has not had recent activity. Tagging @shailesh-vaidya
from cortx-re.
No issues found for last 2 weeks closing this.
from cortx-re.
Related Issues (20)
- CVE-2022-36087 (Medium) detected in oauthlib-3.2.1-py3-none-any.whl - autoclosed HOT 7
- Test Board-Genius Sync HOT 2
- CVE-2022-24439 (Critical) detected in GitPython-3.1.29-py3-none-any.whl HOT 1
- CVE-2022-23491 (Medium) detected in certifi-2022.9.24-py3-none-any.whl - autoclosed HOT 1
- CVE-2022-40897 (Medium) detected in setuptools-36.0.1-py2.py3-none-any.whl HOT 1
- CVE-2023-23931 (Medium) detected in cryptography-3.3.2-cp36-abi3-manylinux2010_x86_64.whl HOT 1
- CVE-2023-0286 (High) detected in cryptography-3.3.2-cp36-abi3-manylinux2010_x86_64.whl HOT 1
- CVE-2023-25577 (High) detected in Werkzeug-2.2.2-py3-none-any.whl HOT 1
- CVE-2023-23934 (Low) detected in Werkzeug-2.2.2-py3-none-any.whl HOT 1
- CVE-2023-30861 (High) detected in Flask-1.1.1-py2.py3-none-any.whl HOT 1
- CVE-2023-40267 (Critical) detected in GitPython-3.1.29-py3-none-any.whl HOT 1
- Combine NODECLI and OPTIMIZED HW Deployment HOT 4
- CVE-2021-33503 (High) detected in urllib3-1.25.11-py2.py3-none-any.whl HOT 16
- cortx-all image generation instructions for community. HOT 3
- BG test HOT 4
- CVE-2021-21330 (Medium) detected in aiohttp-3.6.1-cp37-cp37m-manylinux1_x86_64.whl - autoclosed HOT 3
- CVE-2021-41496 (High) detected in numpy-1.21.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl - autoclosed HOT 3
- CVE-2021-34141 (Medium) detected in numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl HOT 10
- CVE-2022-24302 (Medium) detected in paramiko-2.7.1-py2.py3-none-any.whl - autoclosed HOT 5
- Need to change our template asking for new third party software HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cortx-re.