Comments (26)
hh
commented on August 25, 2024
@Zilman do you have same Azure creds I could use?
Cheers @hh
from demo.
namliz
commented on August 25, 2024
@hh don't have credits for Azure on hand but I can try and get us some, meanwhile you get $200 for free at first I believe.
from demo.
dankohn
commented on August 25, 2024
Gene, can you please give them admin access to the Azure account we setup.
…On Tue, Feb 28, 2017 at 10:40 AM, Eugene ***@***.***> wrote:
@hh <https://github.com/hh> don't have credits for Azure on hand but I
can try and get us some. You get $200 for free at first I believe.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#126 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AC8MBpHnU1I1-0lDinrqP_BBmhhg78Itks5rhD_tgaJpZM4KXHVt>
.
--
Dan Kohn <mailto:[email protected]>
Executive Director, Cloud Native Computing Foundation <https://cncf.io/>
tel:+1-415-233-1000
from demo.
namliz
commented on August 25, 2024
@dankohn I can't, only ever used my personal free account (I think), not an admin on the main one - at least that's what it says when I go to the "Active Directory".
from demo.
hh
commented on August 25, 2024
Working in branch https://github.com/cncf/demo/tree/azure
from demo.
hh
commented on August 25, 2024
We had to migrate to a pay-as-you-go to get around as the free limits on cores is too low for quick-iterations on dev work. We will refactor for smaller instances later to allow free-tier to work after everything is functioning.
from demo.
hh
commented on August 25, 2024
We seem to have triggered block on our API access from NZ.
(az login and API access seems to hit a firewall somewhere)
We moved to an IP in the states for now.
Is there someone at Microsoft that might be interested in supporting / unblocking us as we move along?
from demo.
dankohn
commented on August 25, 2024
I'm hopeful @brendandburns might know someone who could send CNCF a few Azure credits.
from demo.
brendandburns
commented on August 25, 2024
Can you send me the subscription ID (bburns [at] microsoft [dot] com) and I'll see what we can do on this side.
--brendan
from demo.
hh
commented on August 25, 2024
@brendandburns done and thanks!
from demo.
hh
commented on August 25, 2024
We started getting kubelet panics when we started using --cloud-provider=azure
Took a while to come across: kubernetes/kubernetes#42576
Now we're off to generate --cloud-config=azure.json
-- Logs begin at Mon 2017-03-20 19:46:01 UTC, end at Mon 2017-03-20 20:18:32 UTC. --
Mar 20 19:48:56 etcd-master1 systemd[1]: Starting kubelet.service...
Mar 20 19:48:56 etcd-master1 systemd[1]: Started kubelet.service.
Mar 20 19:48:56 etcd-master1 kubelet-wrapper[2257]: + exec /usr/bin/rkt run --volume dns,kind=host,source=/etc/resolv.conf --mount volume=dns,target=/etc/resolv.conf --volume rkt,kind=host,source=/opt/bin/host-rkt --mount volume=rkt,target=/usr/bin/rkt --volume
Mar 20 19:48:59 etcd-master1 kubelet-wrapper[2257]: pubkey: prefix: "quay.io/coreos/hyperkube"
Mar 20 19:48:59 etcd-master1 kubelet-wrapper[2257]: key: "https://quay.io/aci-signing-key"
...
Mar 20 19:48:59 etcd-master1 kubelet-wrapper[2257]: Downloading signature: 473 B/473 B
Mar 20 19:49:00 etcd-master1 kubelet-wrapper[2257]: Downloading ACI: 0 B/237 MB
...
Mar 20 19:49:07 etcd-master1 kubelet-wrapper[2257]: Downloading ACI: 237 MB/237 MB
Mar 20 19:49:37 etcd-master1 kubelet-wrapper[2257]: image: signature verified:
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: panic: runtime error: invalid memory address or nil pointer dereference [recovered]
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: panic: runtime error: invalid memory address or nil pointer dereference
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: [signal 0xb code=0x1 addr=0x20 pc=0xa32559]
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: goroutine 1 [running]:
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: panic(0x448ae60, 0xc820030060)
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: /usr/local/go/src/runtime/panic.go:481 +0x3e6
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: io/ioutil.readAll.func1(0xc820acca40)
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: /usr/local/go/src/io/ioutil/ioutil.go:30 +0x11e
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: panic(0x448ae60, 0xc820030060)
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: /usr/local/go/src/runtime/panic.go:443 +0x4e9
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: bytes.(*Buffer).ReadFrom(0xc820acc998, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: /usr/local/go/src/bytes/buffer.go:176 +0x239
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: io/ioutil.readAll(0x0, 0x0, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: /usr/local/go/src/io/ioutil/ioutil.go:33 +0x156
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: io/ioutil.ReadAll(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: /usr/local/go/src/io/ioutil/ioutil.go:42 +0x51
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: k8s.io/kubernetes/pkg/cloudprovider/providers/azure.NewCloud(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 20 19:50:59 etcd-master1 kubelet-wrapper[2257]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure.go:74 +0x81
Mar
from demo.
namliz
commented on August 25, 2024
Oh boy, on AWS all the stuff it wants in azure.json was simply inferred by Kubernetes (you'd just tag those resources with the cluster name and pass just that to it). This makes it much more convoluted.
This is an unfortunate discrepancy. As far as I know Azure also has the concept of tags.
from demo.
hh
commented on August 25, 2024
Looks like we need to populate this manually:
https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure.go#L38
// Config holds the configuration parsed from the --cloud-config flag
// All fields are required unless otherwise specified
type Config struct {
// The cloud environment identifier. Takes values from https://github.com/Azure/go-autorest/blob/ec5f4903f77ed9927ac95b19ab8e44ada64c1356/autorest/azure/environments.go#L13
Cloud string `json:"cloud" yaml:"cloud"`
// The AAD Tenant ID for the Subscription that the cluster is deployed in
TenantID string `json:"tenantId" yaml:"tenantId"`
// The ID of the Azure Subscription that the cluster is deployed in
SubscriptionID string `json:"subscriptionId" yaml:"subscriptionId"`
// The name of the resource group that the cluster is deployed in
ResourceGroup string `json:"resourceGroup" yaml:"resourceGroup"`
// The location of the resource group that the cluster is deployed in
Location string `json:"location" yaml:"location"`
// The name of the VNet that the cluster is deployed in
VnetName string `json:"vnetName" yaml:"vnetName"`
// The name of the subnet that the cluster is deployed in
SubnetName string `json:"subnetName" yaml:"subnetName"`
// The name of the security group attached to the cluster's subnet
SecurityGroupName string `json:"securityGroupName" yaml:"securityGroupName"`
// (Optional in 1.6) The name of the route table attached to the subnet that the cluster is deployed in
RouteTableName string `json:"routeTableName" yaml:"routeTableName"`
// (Optional) The name of the availability set that should be used as the load balancer backend
// If this is set, the Azure cloudprovider will only add nodes from that availability set to the load
// balancer backend pool. If this is not set, and multiple agent pools (availability sets) are used, then
// the cloudprovider will try to add all nodes to a single backend pool which is forbidden.
// In other words, if you use multiple agent pools (availability sets), you MUST set this field.
PrimaryAvailabilitySetName string `json:"primaryAvailabilitySetName" yaml:"primaryAvailabilitySetName"`
// The ClientID for an AAD application with RBAC access to talk to Azure RM APIs
AADClientID string `json:"aadClientId" yaml:"aadClientId"`
// The ClientSecret for an AAD application with RBAC access to talk to Azure RM APIs
AADClientSecret string `json:"aadClientSecret" yaml:"aadClientSecret"`
}from demo.
hh
commented on August 25, 2024
Yea, would be nice to have this for Azure:
https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/tags.go#L54
// ClusterID is our cluster identifier: we tag AWS resources with this value,
// and thus we can run two independent clusters in the same VPC or subnets.
// This gives us similar functionality to GCE projects.
ClusterID stringfrom demo.
namliz
commented on August 25, 2024
Yes, you have no choice -- and that is not how the other providers are implemented:
https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go#L398
Could be an interesting thing to suggest upstream.
Edit: Jinx. :)
I haven't read the azure provider, do you think it would be messy to add this to it?
from demo.
brendandburns
commented on August 25, 2024
@hh Please see https://github.com/Azure/acs-engine
I think you will find it a much more pleasant way to turn up an Azure kubernetes cluster.
I'll work on the subcription stuff.
from demo.
hh
commented on August 25, 2024
We have been looking pretty heavily at the individual config generation parts at https://github.com/Azure/acs-engine/tree/master/parts
They've been useful in integrating the acs-engine approach.
from demo.
hh
commented on August 25, 2024
Just starting multiple build/deploys at once on Azure, had to increase default core quota from 10 to 100. Should be able to do least ten concurrent builds soon.
from demo.
colemickens
commented on August 25, 2024
RE: Azure relying on hostname == nodeName, this is due to lack of metadata service, so we make assumptions: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_instances.go#L98
from demo.
hh
commented on August 25, 2024
Great to meet you in Berlin @colemickens !
We really are cutting ourselves on some of the bleeding edges of kubernetes + azure + terraform on this one. :)
Let me know if you want some help in creating those upstream issues for azure / kubernetes we talked about here: https://github.com/cncf/demo/blob/fd21acc7655e849a8cdda9faf0c547fa2916a0dc/azure/readme.org#notable-issues
I opened issues for terraform azurerm_dns_srv_record list and azure-sdk-for-go NetworkInterfaceDnsSettings.dnsServers resolution
We'll look into refactoring this sometime soon after another cloud or two and likely add a second Azure approach using acs-engine provisioning of the kubernetes cluster.
from demo.
colemickens
commented on August 25, 2024
-
Azure dns_zones do not provide IPs - Please create a new issue on this repository so that I can get the appropriate folks into a thread with you. Otherwise I fear if I just take this internal I'm going to wind up being a middle-man between you and our folks.
-
Azure CNAME records don’t resolve correctly - Same as the previous one. Need a bit more details on this too - an exact scenario they can repro.
-
Terraform azurerm_dns_srv_records do not support multiple dynamic entries - I think this is the one you opened on Terraform already, if not, please do.
-
Azure Cluster-Autoscale Virtual Machine Scale Sets are not yet supported by kubernetes - We can't do anything about this until the platform grows support for disk attachment to nodes. I've been asking for it for a long time. Unfortunately, no ETA to share now. There are already upstream issue(s) for it.
-
Starting kubelet without –cloud-config=azure.json results in a panic - Please file a bug upstream, should be a quick fix...
-
When using –cloud-provider=azure not only must you use –cloud-config=azure.json, it seems you have to provide all the optional settings as well. Failure to do so results in a panic. - There are no optional settings in that file, I'd roll it into the same issue for the above bullet point.
Finally, I think you might have mentioned another issue where the Azure DNS nameservers weren't forwarding requests? Again, if you can file an Issue against cncf/demo and tag me in it, I will share all three with the right folks internally and get the discussion going.
from demo.
colemickens
commented on August 25, 2024
@hh, I've got an internal mail drafted, just waiting for cncf/demo issues for the other one (or two) issues listed the previous post. Then I can start getting the right folks to chime in. Thanks!
from demo.
hh
commented on August 25, 2024
Seeing that private DNS zones are not yet supported, I suspect that's why CNAME record resolution is broken.
from demo.
hh
commented on August 25, 2024
I've been spinning up AWS and Azure side by side, and I can confirm that the reason we needed a work around for CNAME was due to Azure not yet providing support for private zones
Our work around was to use multiple A records.
We have a working Azure deploy for now, even if we are abusing the public DNS service records. :)
from demo.
discordianfish
commented on August 25, 2024
@colemickens I just ran into panic-when-optional-config-keys-missing issue. Did you fill an issue for that? Can't find it.
from demo.
colemickens
commented on August 25, 2024
@discordianfish I just filed: kubernetes/kubernetes#47543
from demo.
Related Issues (20)
- Doc: Two links to graphics in README.md broken
- Grafana roundtrip json export/import bug HOT 2
- Integrate CoreDNS HOT 4
- Integrate GRPC HOT 1
- Integrate LinkerD HOT 6
- Prometheus resource usage "observer effect"
- Please replace minion terminology
- Port demo to OpenStack
- Azure dns_zones do not provide IPs HOT 3
- Phabricator on Cloud Native Landscape Project poster HOT 2
- GCE Support HOT 1
- Image builds should use ansible from upstream kubernetes HOT 4
- Locate upstream CI binary releases for CNCF Projects HOT 3
- Packet - Bare Metal Cloud HOT 6
- Integrate Bluemix HOT 3
- Controlling the BOINC client via RPC
- e2e CI setup (WIP) HOT 2
- GKE
- Azure Container Service Engine
- Adjust language about CNCF Community cluster in overview HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from demo.