symphoniacloud / lambda-monitoring Goto Github PK
View Code? Open in Web Editor NEWLogging and metrics libraries for AWS Lambda
License: Apache License 2.0
Logging and metrics libraries for AWS Lambda
License: Apache License 2.0
I've added the symphonia 1.0.3 dependency to my project and have been experiencing problems logging stack traces when running locally (not in the lambda context):
Here is the java code:
package com.test;
import lombok.extern.slf4j.Slf4j;
@Slf4j
public class MainTest {
public static void main(String[] args) {
log.error("error", new CustomException());
}
static class CustomException extends RuntimeException {
private static final long serialVersionUID = 1L;
}
}
This is what i get on STDOUT:
at com.test.MainTest.main(MainTest.java:8)
Normal log.* calls without Throwables are fine (they arrive on STDOUT), it's when logging using the signature that takes a String message and a Throwable object that i get a problem.
Has anyone else experienced this?
Sorry for the confusion, but it's not clear to me on the Readme if this library can be used a logger implementation for SLF4J.
I have a library that uses SLF4J as a logger, if I add lambda-logging
to my project, will all 3rd party library invocations to SLF4J be routed to lambda-logging
automatically?
Hello, I use java.util.concurrent.ForkJoinPool to run a async task in a AWS Lambda. Unfortunately the the logs out of that Thread have no request ID. The other logs from the main Thread have request ids.
[2020-07-02 14:34:17.868] NO_REQUEST_ID INFO d.g.m.l.DfiConnector:107 - Sending message
[2020-07-02 14:34:18.750] NO_REQUEST_ID INFO d.g.m.l.DfiConnector:128 - responded OK
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="STDOUT" class="io.symphonia.lambda.logging.DefaultConsoleAppender">
<encoder>
<pattern>[%date{yyyy-MM-dd HH:mm:ss.SSS}] %X{AWSRequestId:-NO_REQUEST_ID} %.-6level %logger{5}:%line - %replace(%msg){'\n','
'} %replace(%exception){'\n','
'} %nopexception %n</pattern>
</encoder>
</appender>
....
<root level="${loglevelroot:-INFO}">
<appender-ref ref="STDOUT"/>
</root>
</configuration>
io.symphonia:lambda-logging:1.0.3
The AWS Lambda docs says the aws-lambda-java-core
is already into the machine running the Lambda, so it's an optimization to exclude the library from the package as:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.1.0</version>
<scope>provided</scope>
</dependency>
I tried to use the lib and it is not picking the request id.
I can't see any functional reason to use @CloudWatchLogGroup since Lambda controls where the logs go. Your implementation can simply infer the log group to be "/aws/lambda/{functionName}" where {functionName} is the value of AWS_LAMBDA_FUNCTION_NAME environment variable.
I'm trying to avoid using different sets of logging dependencies when running my code as part of local test run, and later in production running in a lambda context.
So here is what i did:
I've added the lambda-logging dependency to my project
<dependency>
<groupId>io.symphonia</groupId>
<artifactId>lambda-logging</artifactId>
<version>1.0.2</version>
</dependency>
Then used a log.info in one of my test cases, to no effect.
Edit: The io.symphonia.lambda.logging.DefaultLogbackConfigurator#configure is not invoked
The io.symphonia.lambda.logging.DefaultLogbackConfigurator#configure is invoked (I checked that in the debugger),
but nothing shows in STDOUT.
The io.symphonia.lambda.logging.DefaultConsoleAppender DEFAULT_PATTERN is ready for the case where there is no AWSRequestID:
%X{AWSRequestId:-" + NO_REQUEST_ID + "}
So it stands to reason that adding the lambda-logging dependency should be all i need in order to have logging locally and in a lambda context.
What am I missing?
lambda-metrics looks like a great solution for custom metrics in Lambda, thanks for this!
One small suggestion: Many teams deploy project infrastructure via CloudFormation, so generating CFN template for metrics filters instead of creating them at build time would probably fit better into many workflows (and integrate better with SAM).
Cheers,
Ryan
lambda-monitoring/lambda-logging uses logback-classic 1.1.8 which is vulnerable to CVE-2017-5929.
Per the note at https://logback.qos.ch/news.html dated Feb 8th, 2017:
"Release 1.2.0 fixes a rather severe serialization vulnerability in SocketServer and ServerSocketReceiver. Users running these components should upgrade immediately."
It is advised to upgrade the logback dependency to 1.2.0 or later.
I'm using the custom configuration, as showed in the lambda logging readme.
This is my logback.xml file:
<configuration>
<appender name="STDOUT" class="io.symphonia.lambda.logging.LambdaConsoleAppender">
<encoder>
<pattern>{"processId": "%X{AWSRequestId:-" + NO_REQUEST_ID + "}","datetime":"%d{yyyy-MM-dd HH:mm:ss.SSS}","level":"%.-6level","logger":"%logger{5}","message":"%msg"} } %n</pattern>
</encoder>
</appender>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</configuration>
The log will have all log lines into one, ignoring the %n line separator that I've put there. My output will be a json.
Any ideas about this problem?
The same error reported here is happening on this library as well. The aws-lambda-java-log4j2 team fixed it, so you can copy their code I guess.
The problem is, since you are simply printing the log to the console, strings with \n
will be printed as new lines in CloudWatch
, this becomes REALLY troublesome when we try to log Exceptions
, since it will be print as 1 log entry per line of stack trace, which makes our logs in CloudWatch a complete mess.
This is my test code:
Exception exception = new IllegalArgumentException(new NullPointerException("test exception"));
LOGGER.info("test 1\n test 2 \n test 3");
LOGGER.info("test 4");
LOGGER.error("new error test", exception);
And this is the result:
I got this message in my CloudWatch Logs while using version 1.0.3, probably from another library that uses SLF4J with a newer version?
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#noProviders for further details.
SLF4J: Class path contains SLF4J bindings targeting slf4j-api versions prior to 1.8.
SLF4J: Ignoring binding found at [jar:file:/var/task/lib/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#ignoredBindings for an explanation.
Thanks for the brilliant library and it works great.
Is it possible to log a certain variable from code in it?
e.g. when my lambda is invoked from S3 file upload. I need to print the file name in all log statements.
Thanks
Is it possible to change the logging level via the application.properties
by doing something like logging.level.com.package:LEVEL
?
I have been trying to use the lambda-logging library and I noticed something very odd - when running my lambda's using SAM local - the logs were not showing up at all - but when ran on AWS as a real lambda, they did show up.
What bothered me even more though is a bit hard to describe. Only the SLF4J impl seems to fail. Given I have the following in my handler:
lambdaLogger.log("using the lambda logger y'all...")
System.err.println("A standard error output...")
logger.error("Using SLF4J - first call")
logger.error("Using SLF4j - second call")
On SAM Local I receive the following (note the ordering of events and that SLF4J is missing)
START RequestId: b280dc8f-2a62-4131-b023-802fdf4222e8 Version: $LATEST
using the lambda logger y'all...A standard error output...
END RequestId: b280dc8f-2a62-4131-b023-802fdf4222e8
REPORT RequestId: b280dc8f-2a62-4131-b023-802fdf4222e8 Duration: 902.05 ms Billed Duration: 1000 ms Memory Size: 3008 MB Max Memory Used: 3 MB
On AWS I receive the following, this time with SLF4J logs being printed (note the ordering of events -the SLF4J's calls are AFTER the END RequestId
) and the two different log statements got grouped into a single one. I thought they should have been two unique events since they were two individual calls to logger.error
?
Seeing how the SLF4J calls only show up when running on AWS is odd but when also seeing them appear out of order makes me wonder if SLF4J is doing some sort of queuing under the covers even though I don't believe the ConsoleAppender is an async appender. What I'm wondering if is this is causing a dangerous race condition...the logs didn't actually get written out to the STDOUT synchronously with where the logs were told to print in the code before continuing to the next statement in the code.... just by chance, the real lambda is still getting them written before the function container is paused / killed whereas when using SAM local the function container is killed before they are written which is why I am not seeing them. Does that sound plausible?
Finally - I am just wondering y'alls opinion going forward now that AWS has a log4j2 appender (https://github.com/aws/aws-lambda-java-libs/tree/master/aws-lambda-java-log4j2) whereas it was just an old 1.x appender when y'all first wrote this library (I believe that's what I understood from y'alls blog post). I also found it interesting that in their log4j2 impl they just call out to the lambda logger: https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-log4j2/src/main/java/com/amazonaws/services/lambda/runtime/log4j2/LambdaAppender.java#L74 I saw y'alls note in the README about possibly needing to make this SLF4J impl call out to the lambdalogger directly: https://github.com/symphoniacloud/lambda-monitoring/blob/master/lambda-logging/README.md#todo .. do you think that is necessary to get around the issues above? ^^
Can you please provide an example for this in the docs? I've checked out a couple of examples on Stackoverflow but don't know how to adapt them to this library:
https://stackoverflow.com/questions/9116425/apache-log4j-logging-with-specific-timezone
https://stackoverflow.com/questions/1785725/specify-time-zone-of-log4js-date
https://stackoverflow.com/questions/32090902/why-does-log4j-message-produce-timestamps-in-utc
I tried this syntax:
<pattern>%d{ISO8601}{GMT-5} %p [%c] - %m%n [%d{yyyy-MM-dd HH:mm:ss.SSS}] %X{AWSRequestId:-" + NO_REQUEST_ID + "} %.-6level %logger{5} - %msg \r%replace(%ex){'\n','\r'}%nopex</pattern>
But it outputs 2018-06-21 12:28:36,742{GMT-5}
. I also tried adding the Log4J extras to the classpath by including it in my pom.xml https://mvnrepository.com/artifact/log4j/apache-log4j-extras/1.2.17 but that didn't help either.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.