Comments (37)
people are reporting that the API is working again without crumb. Can you confirm this?
from ticker.sh.
Here's the output:
$ curl -v "https://finance.yahoo.com" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8" -H 'cookie: notice_behavior=expressed,eu; notice_gdpr_prefs=0,1,2:1a8b5228dd7ff0717196863a5d28ce6c;'
* Trying 87.248.119.252:443...
* Connected to finance.yahoo.com (87.248.119.252) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, CERT verify (15):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
* subject: C=US; ST=California; L=Sunnyvale; O=Oath Holdings Inc.; CN=*.api.fantasysports.yahoo.com
* start date: Apr 10 00:00:00 2023 GMT
* expire date: May 31 23:59:59 2023 GMT
* subjectAltName: host "finance.yahoo.com" matched cert's "*.yahoo.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: finance.yahoo.com]
* h2h3 [user-agent: curl/7.87.0]
* h2h3 [accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8]
* h2h3 [cookie: notice_behavior=expressed,eu; notice_gdpr_prefs=0,1,2:1a8b5228dd7ff0717196863a5d28ce6c;]
* Using Stream ID: 1 (easy handle 0x5561bae30080)
> GET / HTTP/2
> Host: finance.yahoo.com
> user-agent: curl/7.87.0
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
> cookie: notice_behavior=expressed,eu; notice_gdpr_prefs=0,1,2:1a8b5228dd7ff0717196863a5d28ce6c;
>
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 307
< date: Fri, 21 Apr 2023 16:42:59 GMT
< strict-transport-security: max-age=31536000
< server: ATS
< cache-control: no-store
< content-type: text/html; charset=utf-8
< content-language: en
< expect-ct: max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only"
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< set-cookie: notice_behavior=DELETE; Expires=Fri, 21 Apr 2023 16:32:59 GMT; Max-Age=0; Domain=finance.yahoo.com; Path=; Secure; HttpOnly
< set-cookie: notice_behavior=DELETE; Expires=Fri, 21 Apr 2023 16:32:59 GMT; Max-Age=0; Domain=yahoo.com; Path=; Secure; HttpOnly
< set-cookie: notice_gdpr_prefs=DELETE; Expires=Fri, 21 Apr 2023 16:32:59 GMT; Max-Age=0; Domain=finance.yahoo.com; Path=; Secure; HttpOnly
< set-cookie: notice_gdpr_prefs=DELETE; Expires=Fri, 21 Apr 2023 16:32:59 GMT; Max-Age=0; Domain=yahoo.com; Path=; Secure; HttpOnly
< location: https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=Rxwfl3c&done=https%3A%2F%2Ffinance.yahoo.com%2F
< set-cookie: GUCS=AUccH5d3; Max-Age=1800; Domain=.yahoo.com; Path=/; Secure
< content-length: 0
<
* Connection #0 to host finance.yahoo.com left intact
(It's from another machine with another ISP, but still in the same general area).
I confirm that API is working without the crumb, I reported that a few posts back ;) I just checked and it still works.
from ticker.sh.
Happily reverted this change!
I confirm that API is working without the crumb, I reported that a few posts back ;) I just checked and it still works.
@tomaszg7 I ignored your remark at that time, as I observed the API experiencing intermittent issues for days, if not weeks, before it eventually completely broke down. I thought that these disruptions might have been due to A/B testing or similar activities, and I didn't expect it to recover. It seems the joke's on me this time.
If the issue reoccurs, I would be grateful to reach out to both you and @servingbaby for assistance. Thank you for your collaboration!
from ticker.sh.
nevermind, script is not broken. appears to have something to do with the rare google outage that happened today
from ticker.sh.
I am getting the same error trying to use the script.
jq: error (at <stdin>:1): Cannot iterate over null (null)
Does it work for you or does it keep giving the error?
from ticker.sh.
from ticker.sh.
Yahoo changed something in their finance API. Tried to debug it, but there’s no quick fix.
Basically, they now expect another query parameter which is available from within a browser session. Will need to dig into it a little deeper.
from ticker.sh.
Here’s how other people are handling this. Two extra calls to obtain a session and fetch a crumb.
Implementation for us would look something like this: If crumb is unavailable or API returns 401, obtain new session, fetch crumb (store it in temporary file for subsequent requests) and repeat the initial call to fetch quotes.
from ticker.sh.
Please test this and let me know whether it works and which OS you're using: #35
I got this error at the moment, though I do have this running with root via /usr/local/sbin/:
"mktemp: too few X's in template 'pstadler-ticker-sh'"
Linux?
from ticker.sh.
Please test this and let me know whether it works and which OS you're using: #35
I got this error at the moment, though I do have this running with root via /usr/local/sbin/:
"mktemp: too few X's in template 'pstadler-ticker-sh'"Linux?
Yes, I'm currently on Arch :)
from ticker.sh.
So what I‘m ultimately trying to do is to persist sessions across executions. #35 works just fine on macOS, but mktemp requires some sorts of random bits which is quite annoying. I should have some time later tonight to work out a solution that works across platforms.
from ticker.sh.
I can confirm working well on my particular environment. Great work and Thank you! @pstadler
from ticker.sh.
merged into master. please pull the latest version.
from ticker.sh.
Cookie file:
# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
.yahoo.com TRUE / TRUE 1682020284 GUCS AQOsjLu3
from ticker.sh.
That‘s mighty odd. Thanks for helping out each other here 👍🏻
from ticker.sh.
Tried running curl command without silent flag and nothing weird happens. Here's output
$ curl --output /tmp/yyy --cookie-jar /tmp/xxx "https://finance.yahoo.com" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
The file /tmp/yyy
is empty and /tmp/xxx
is similar to the one I previously shown.
from ticker.sh.
Also tried updating curl
and enabling some extra features, but no change:
curl 8.0.1 (x86_64-pc-linux-gnu) libcurl/8.0.1 OpenSSL/1.1.1t zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 c-ares/1.19.0 libidn2/2.3.4 nghttp2/1.51.0
Release-Date: 2023-03-20
Protocols: dict file ftp ftps http https imap imaps mqtt pop3 pop3s rtsp smtp smtps tftp
Features: AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets zstd
Tried it also on another machine (also a similar Gentoo box, but connected to different ISP) and the result was the same.
from ticker.sh.
from ticker.sh.
I am also getting this error. I think the Yahoo finance API is failing and giving jq bad data.
from ticker.sh.
This appears to be an ongoing issue
from ticker.sh.
Please test this and let me know whether it works and which OS you're using: #35
from ticker.sh.
Oh!
I also just created a quick pull request, testing yours too.
from ticker.sh.
Please test this and let me know whether it works and which OS you're using: #35
I got this error at the moment, though I do have this running with root via /usr/local/sbin/:
"mktemp: too few X's in template 'pstadler-ticker-sh'"
from ticker.sh.
Updated PR, tested with macOS and Arch. This should now work in most environments.
from ticker.sh.
For me (on Gentoo) the old version started to work now (it was broken a few hours ago). New version however doesn't produce any output. No error message, no quotes, nothing.
Edit: here's content of my crumb file:
<!doctype html public "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>Yahoo! - Error report</title>
<style>
/* nn4 hide */ /*/*/
body {
font: small/ 1.2em arial, helvetica, clean, sans-serif;
font: x-small;
text-align: center;
}
table {
font-size: inherit;
font: x-small;
}
html>body {
font: 83%/ 1.2em arial, helvetica, clean, sans-serif;
}
input {
font-size: 100%;
vertical-align: middle;
}
p,form {
margin: 0;
padding: 0;
}
p {
padding-bottom: 6px;
margin-bottom: 10px;
}
#doc {
width: 48.5em;
margin: 0 auto;
border: 1px solid #fff;
text-align: center;
}
#ygma {
text-align: right;
margin-bottom: 53px
}
#ygma img {
float: left;
}
#ygma div {
border-bottom: 1px solid #ccc;
padding-bottom: 8px;
margin-left: 152px;
}
#bd {
clear: both;
text-align: left;
width: 75%;
margin: 0 auto 20px;
}
h1 {
font-size: 135%;
text-align: center;
margin: 0 0 15px;
}
legend {
display: none;
}
fieldset {
border: 0 solid #fff;
padding: .8em 0 .8em 4.5em;
}
form {
position: relative;
background: #eee;
margin-bottom: 15px;
border: 1px solid #ccc;
border-width: 1px 0;
}
#s1p {
width: 15em;
margin-right: .1em;
}
form span {
position: absolute;
left: 70%;
top: .8em;
}
form a {
font: 78%/ 1.2em arial;
display: block;
padding-left: .8em;
white-space: nowrap;
background: url(https://s.yimg.com/lq/a/i/s/bullet.gif)
no-repeat left center;
}
form .sep {
display: none;
}
.more {
text-align: center;
}
#ft {
padding-top: 10px;
border-top: 1px solid #999;
}
#ft p {
text-align: center;
font: 78% arial;
}
/* end nn4 hide */
</style>
</head>
<body>
<div id="doc">
<div id="ygma">
<a href="http://www.yahoo.com">
<img src="https://s.yimg.com/lq/a/i/yahoo.gif" width="147"
height="31" border="0" alt="Yahoo!">
</a>
<div>
<a href="http://www.yahoo.com">Yahoo!</a>
- <a href="http://help.yahoo.com">Help</a>
</div>
</div>
<div id="bd">
<h1>HTTP Status 500 - Server Error</h1>
<p>
<b>type</b>
Exception report
</p>
<p>
<b>message</b>
<u>Server Error</u>
</p>
<p>
<b>description</b>
<u>Server Error</u>
</p>
<p>
</p>
</div>
</div>
</body>
</html>
from ticker.sh.
For me (on Gentoo) the old version started to work now (it was broken a few hours ago). New version however doesn't produce any output. No error message, no quotes, nothing.
Edit: here's content of my crumb file:
<!doctype html public "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> ... ```
I wonder if its giving a 500 cause of the cookie file? Does that get generated okay?
from ticker.sh.
Yep, I think it might be that,
I was able to recreate your error with your cookies file, but my file worked okay:
$ curl --silent -b cookies.txt "https://query1.finance.yahoo.com/v1/test/getcrumb"
500 Error
vs
$ curl --silent -b cookies.txt.old "https://query1.finance.yahoo.com/v1/test/getcrumb"
fk22Yb6tiAY
However, my cookies.txt file has a lot more content in there, for example (removing what might be sensitive):
# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
.yahoo.com TRUE / TRUE 0 A1S d=-----
#HttpOnly_.yahoo.com TRUE / TRUE 1713575851 A3 d=-------
#HttpOnly_.yahoo.com TRUE / TRUE 1713575851 A1 d=-------
I am hoping this helps the dev :)
from ticker.sh.
The HttpOnly cookies are missing. what‘s your curl version?
from ticker.sh.
curl 7.88.1 (x86_64-pc-linux-gnu) libcurl/7.88.1 OpenSSL/1.1.1t zlib/1.2.13 brotli/1.0.9 c-ares/1.19.0 nghttp2/1.51.0
Release-Date: 2023-02-20
Protocols: dict file ftp ftps http https imap imaps mqtt pop3 pop3s rtsp smtp smtps tftp
Features: AsynchDNS brotli HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets
from ticker.sh.
Here's mine for context:
$ curl --version
curl 7.87.0 (x86_64-pc-linux-gnu) libcurl/7.87.0 OpenSSL/3.0.7 zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.10.0 nghttp2/1.51.0
Release-Date: 2022-12-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd
Edit: went ahead and updated, and still working okay for me with:
$ curl --version
curl 8.0.1 (x86_64-pc-linux-gnu) libcurl/8.0.1 OpenSSL/3.0.7 zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.10.0 nghttp2/1.51.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd
from ticker.sh.
This one is a real head scratcher for me
from ticker.sh.
@tomaszg7 please run this and paste the result:
curl -v "https://finance.yahoo.com" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"
from ticker.sh.
* Trying 87.248.106.204:443...
* Connected to finance.yahoo.com (87.248.106.204) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
* subject: C=US; ST=California; L=Sunnyvale; O=Oath Holdings Inc.; CN=*.api.fantasysports.yahoo.com
* start date: Apr 10 00:00:00 2023 GMT
* expire date: May 31 23:59:59 2023 GMT
* subjectAltName: host "finance.yahoo.com" matched cert's "*.yahoo.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: finance.yahoo.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8]
* Using Stream ID: 1 (easy handle 0x55e9eda3d050)
> GET / HTTP/2
> Host: finance.yahoo.com
> user-agent: curl/7.88.1
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 307
< date: Fri, 21 Apr 2023 06:31:42 GMT
< strict-transport-security: max-age=31536000
< server: ATS
< cache-control: no-store
< content-type: text/html; charset=utf-8
< content-language: en
< expect-ct: max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only"
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< location: https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=BlQKcX8&done=https%3A%2F%2Ffinance.yahoo.com%2F
< set-cookie: GUCS=AQZUCnF_; Max-Age=1800; Domain=.yahoo.com; Path=/; Secure
< content-length: 0
<
* Connection #0 to host finance.yahoo.com left intact
from ticker.sh.
that’s it, you’re getting redirected:
< location: https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=BlQKcX8&done=https%3A%2F%2Ffinance.yahoo.com%2F
from ticker.sh.
Very interesting!
Something about a consent page for GDPR consent. I wonder if its a geo-location thing that you might be running in-to that present that redirect..
Out of my comfort zone here, but there might be a way to inject a header to accept the consent cookie and then continue on with the call.
Would you be able to give this a try? :
curl -v "https://finance.yahoo.com" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8" -H 'cookie: notice_behavior=expressed,eu; notice_gdpr_prefs=0,1,2:1a8b5228dd7ff0717196863a5d28ce6c;'
Additional Ref:
https://stackoverflow.com/questions/64912909/getting-consent-cookies-for-website-login-using-python-requests-cmp-euconsent-v
https://adinserter.pro/faq/gdpr-compliance-cookies-consent
from ticker.sh.
people are reporting that the API is working again without crumb. Can you confirm this?
I am able to confirm too :)
commented out the following to test:
CRUMB_FILE="${SESSION_DIR}/crumb.txt"
curl --silent -b "$COOKIE_FILE" "https://query1.finance.yahoo.com/v1/test/getcrumb" \
"$CRUMB_FILE"
and adjusted to this:
fetch_quotes () { curl --silent -b "$COOKIE_FILE" "$API_ENDPOINT&fields=$fields&symbols=$symbols" }
Edit: So odd, I also removed the cookie and it works like that too lol
fetch_quotes () {
#curl --silent -b "$COOKIE_FILE" "$API_ENDPOINT&fields=$fields&symbols=$symbols"
to
curl --silent "$API_ENDPOINT&fields=$fields&symbols=$symbols"
}
from ticker.sh.
Well, apparently we're back to square one. The script is again broken. Maybe they'll fix it once more in a day or two...
from ticker.sh.
Related Issues (20)
- No-color option for the ouput
- [Feature Request] Implementing Sparklines HOT 4
- Symbol listed twice crashes ticker.sh script HOT 1
- Symbol with "-" are not supported HOT 2
- Upgrade request - Display personal gains/ loose for each symbol HOT 2
- a couple of simple improvements HOT 2
- What does the * mean on the output? HOT 1
- problem with floating point conversion HOT 4
- Feature request, ability to change currency. HOT 1
- What does the * symbol at the end indicate ? HOT 1
- Add `date` to output HOT 1
- Script stopped working HOT 2
- stopped working jq: error (at <stdin>:1): Cannot iterate over null (null) HOT 3
- 401 Unauthorized / 404 Not Found / Cannot iterate over null HOT 28
- missing regularMarketChangePercent is not handled HOT 1
- Too Many Requests HOT 5
- Too Many Requests in crumb.txt, no output from script HOT 5
- stopped working again HOT 35
- python? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ticker.sh.