swiftype-demo / orchestrator Goto Github PK

Dev server implemented to poll advertService dependencies to address issues with marketing upload of bad images. Currently polling production every 10s

Installed: Mar 17, 2022
Cloud provider: aws
Region: us-east-1
Instance ID: i-0eb9d25d708f248e7
OS: Debian 10
Language: java

modified spring boot prod app deployed for polling issues

I noticed in some logging the following errors:

2017-06-12 02:33:09 ERROR dial tcp 10.1.2.3:3306: connect: cannot assign requested address
2017-06-12 02:33:09 CRITICAL dial tcp 1.1.2.3:3306: connect: cannot assign requested address
2017-06-12 02:33:09 ERROR dial tcp 10.1.2.3:3306: connect: cannot assign requested address

This happened for a number of destination mysql servers that orchestrator was expected to discover.

It seems that the cause of this is that I’ve run out of local ports to make a tcp connection.

I am not sure how orchestrator treats this error but this error is NOT a remote server error but a local network (congestion) problem. I think it is likely that orchestrator may consider this a remote server failure, which if incorrect might lead it to try to recover the server when actually it is a local problem. In fact I did notice orchestrator reporting a number of issues last night related to this (output below is from an alerting interface but the output has been somewhat anonymised):

02:34 : orchestrator [critical]: server1 PROBLEM DETECTED: Intermediate master cannot be reached by orchestrator and all of its replicas are unreachable
02:34 : orchestrator [critical]: server2 PROBLEM DETECTED: Master cannot be reached by orchestrator and none of its replicas is replicating
02:34 : orchestrator [critical]: server3 PROBLEM DETECTED: Intermediate master cannot be reached by orchestrator and all of its replicas are unreachable

My thoughts are:

• if this is considered as a remote server problem that should be changed
• it might be convenient to generate a counter which holds the number of times this happens so we can monitor it.
• it may be worth adding a documentation note on this problem and provide some pointers as to what it is and provide suggestions on how to solve it. Probably some generic comments are sufficient.

Hi team,
I've been working on the PaymentService checkout process today and noticed a few issues with it. I'm getting lots of dropped connections and repeated 503' errors.
Is this something on our end?
The dummy credit card works for all all other checkout tests and we don't verify it with the bank for tests, so that's not the issue.
Hi team,
I've been working on testing Stealth Shield and noticed a few issues with it recently. I'm getting lots of dropped connections and repeated 503's (logs attached)

[2021-05-24T11:25:54+0000] DEBUG "PUT /api/[email protected]&credit_card_credit_card_number=340000000000009&credit_card_cvv=473&credit_card_expiration_year=2023&credit_card_expiration_month=10&user_id=testuser"
[2021-05-24T11:25:54+0000] INFO "USER testuser authenticated" 
[2021-05-24T11:25:54+0000] INFO "Attempting to checkout
[2021-05-24T11:25:54+0000] ERROR "Failed to install checkout - response from backend was 503." 

Can someone advise what I should do next please?

Tags
java, process, user-agent, http headers, suspicious dependencies

Environments
dev, staging, qa, prod, client, infra

Relevant Alerts:

• suspicious JAVA child process
• Suspected Log4shell activity
• Malware download / execution
• Suspicious user-agent
• Suspicious http headers

Investigation
Search Rule (with APM instrumentation (user-agent)

• !transaction.request.headers.User-Agent.keyword:Elastic-Heartbeat* AND (jndi OR $jndi* OR jndi~1 OR ldap OR :ldap*)

Search Rule (with APM instrumentation (x-serviice-api)

• !transaction.request.headers.x-api-service.keyword:Elastic-Heartbeat* AND (jndi OR $jndi* OR jndi~1 OR ldap OR :ldap*)

OSQuery - Running log4shell vulnerable processes

• WITH target_jars AS ( SELECT DISTINCT path FROM ( WITH split(word, str) AS( SELECT '', cmdline || ' ' FROM processes UNION ALL SELECT substr(str, 0, instr(str, ' ')), substr(str, instr(str, ' ') + 1) FROM split WHERE str != '') SELECT word AS path FROM split WHERE word LIKE '%.jar' UNION ALL SELECT path FROM process_open_files WHERE path LIKE '%.jar' ) ) SELECT path, matches FROM yara WHERE path IN (SELECT path FROM target_jars) AND count > 0 AND sigrule IN ( 'rule log4jJndiLookup { strings: $jndilookup = "JndiLookup" condition: $jndilookup }', 'rule log4jJavaClass { strings: $javaclass = "org/apache/logging/log4j" condition: $javaclass }' );

Orchestrator allows you to determine (if you want) which servers should be preferred if you have a failover scenario and you need to pick a new master.

There's currently no separation of the behaviour of primary and intermediate master failovers.

In some situations it would be good to be able to allow a server to be used for promotion as an intermediate master but not a primary master. This could be due to it running on special hardware, having special storage or being located in a specific datacentre or zone.

The current rules which are allowed for specifying candidate master promotion rules give the following values:
must, prefer, neutral (default), prefer_not (not implemented), must_not.
The situation with a primary master would be that you want something like must_not_be_masterand this property would be independent of the others.

A suggestion discussed with Shlomi but still open for discussion is to allow the 2 settings to be combined so that you might have prefer,must_not_master, or must_not_master (implying neutral,must_not_master). Some settings would be redundant like must_not,must_not_master but could be accepted for "convenience".

This change would allow the user to adjust the promotion rule settings to be more specific and therefore cover more use cases such as those described above.

Hi,
In our configuration we have mult-dc co-masters and only those can have RW traffic all other replicas can serve as RO replicas. Is there a way to cause orchestrator to set all non co-master replicas to be read-only, or is that something I need to set manually ?

BR

This happened to me recently:

1. orchestrator -c relocate-slaves -i mydb -d otherdb -verbose
2. Orchestrator stops slaves and figures out position etc
3. One replica was stuck (many SHOW SLAVE STATUS commands running, etc)
4. After waiting a while I pressed ^C
5. Then I had to run START SLAVE on all replicas

Eventually I restarted MySQL on the broken replica and retried the action. However it would be nice if Orchestrator could take care of item 5 to reduce the time in which replicas are stopped and to reduce manual actions.