安科网

聊聊flink的logback配置

zhangyou00

zhangyou00

2019-06-30

关注 关注

序

本文主要研究一下flink的logback配置

client端pom文件配置

<dependencies>
    <!-- Add the two required logback dependencies -->
    <dependency>
        <groupId>ch.qos.logback</groupId>
        <artifactId>logback-core</artifactId>
        <version>1.2.3</version>
    </dependency>
    <dependency>
        <groupId>ch.qos.logback</groupId>
        <artifactId>logback-classic</artifactId>
        <version>1.2.3</version>
    </dependency>

    <!-- Add the log4j -> sfl4j (-> logback) bridge into the classpath
     Hadoop is logging to log4j! -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>log4j-over-slf4j</artifactId>
        <version>1.7.15</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-java</artifactId>
        <version>1.7.1</version>
        <exclusions>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>*</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-java_2.11</artifactId>
        <version>1.7.1</version>
        <exclusions>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>*</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-clients_2.11</artifactId>
        <version>1.7.1</version>
        <exclusions>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>*</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
</dependencies>
  • 添加logback-core、logback-classic及log4j-over-slf4j依赖，之后对flink-java、flink-streaming-java_2.11、flink-clients_2.11等配置log4j及slf4j-log4j12的exclusions；最后通过mvn dependency:tree查看是否还有log4j12，以确认下是否都全部排除了

服务端配置

  • 添加logback-classic.jar、logback-core.jar、log4j-over-slf4j.jar到flink的lib目录下(比如/opt/flink/lib)
  • 移除flink的lib目录下(比如/opt/flink/lib)log4j及slf4j-log4j12的jar(比如log4j-1.2.17.jar及slf4j-log4j12-1.7.15.jar)
  • 如果要自定义logback的配置的话，可以覆盖flink的conf目录下的logback.xml、logback-console.xml或者logback-yarn.xml

flink-daemon.sh

flink-release-1.7.1/flink-dist/src/main/flink-bin/bin/flink-daemon.sh

#!/usr/bin/env bash
################################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
################################################################################

# Start/stop a Flink daemon.
USAGE="Usage: flink-daemon.sh (start|stop|stop-all) (taskexecutor|zookeeper|historyserver|standalonesession|standalonejob) [args]"

STARTSTOP=$1
DAEMON=$2
ARGS=("${@:3}") # get remaining arguments as array

bin=`dirname "$0"`
bin=`cd "$bin"; pwd`

. "$bin"/config.sh

case $DAEMON in
    (taskexecutor)
        CLASS_TO_RUN=org.apache.flink.runtime.taskexecutor.TaskManagerRunner
    ;;

    (zookeeper)
        CLASS_TO_RUN=org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer
    ;;

    (historyserver)
        CLASS_TO_RUN=org.apache.flink.runtime.webmonitor.history.HistoryServer
    ;;

    (standalonesession)
        CLASS_TO_RUN=org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint
    ;;

    (standalonejob)
        CLASS_TO_RUN=org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint
    ;;

    (*)
        echo "Unknown daemon '${DAEMON}'. $USAGE."
        exit 1
    ;;
esac

if [ "$FLINK_IDENT_STRING" = "" ]; then
    FLINK_IDENT_STRING="$USER"
fi

FLINK_TM_CLASSPATH=`constructFlinkClassPath`

pid=$FLINK_PID_DIR/flink-$FLINK_IDENT_STRING-$DAEMON.pid

mkdir -p "$FLINK_PID_DIR"

# Log files for daemons are indexed from the process ID's position in the PID
# file. The following lock prevents a race condition during daemon startup
# when multiple daemons read, index, and write to the PID file concurrently.
# The lock is created on the PID directory since a lock file cannot be safely
# removed. The daemon is started with the lock closed and the lock remains
# active in this script until the script exits.
command -v flock >/dev/null 2>&1
if [[ $? -eq 0 ]]; then
    exec 200<"$FLINK_PID_DIR"
    flock 200
fi

# Ascending ID depending on number of lines in pid file.
# This allows us to start multiple daemon of each type.
id=$([ -f "$pid" ] && echo $(wc -l < "$pid") || echo "0")

FLINK_LOG_PREFIX="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-${DAEMON}-${id}-${HOSTNAME}"
log="${FLINK_LOG_PREFIX}.log"
out="${FLINK_LOG_PREFIX}.out"

log_setting=("-Dlog.file=${log}" "-Dlog4j.configuration=file:${FLINK_CONF_DIR}/log4j.properties" "-Dlogback.configurationFile=file:${FLINK_CONF_DIR}/logback.xml")

JAVA_VERSION=$(${JAVA_RUN} -version 2>&1 | sed 's/.*version "\(.*\)\.\(.*\)\..*"/\1\2/; 1q')

# Only set JVM 8 arguments if we have correctly extracted the version
if [[ ${JAVA_VERSION} =~ ${IS_NUMBER} ]]; then
    if [ "$JAVA_VERSION" -lt 18 ]; then
        JVM_ARGS="$JVM_ARGS -XX:MaxPermSize=256m"
    fi
fi

case $STARTSTOP in

    (start)
        # Rotate log files
        rotateLogFilesWithPrefix "$FLINK_LOG_DIR" "$FLINK_LOG_PREFIX"

        # Print a warning if daemons are already running on host
        if [ -f "$pid" ]; then
          active=()
          while IFS='' read -r p || [[ -n "$p" ]]; do
            kill -0 $p >/dev/null 2>&1
            if [ $? -eq 0 ]; then
              active+=($p)
            fi
          done < "${pid}"

          count="${#active[@]}"

          if [ ${count} -gt 0 ]; then
            echo "[INFO] $count instance(s) of $DAEMON are already running on $HOSTNAME."
          fi
        fi

        # Evaluate user options for local variable expansion
        FLINK_ENV_JAVA_OPTS=$(eval echo ${FLINK_ENV_JAVA_OPTS})

        echo "Starting $DAEMON daemon on host $HOSTNAME."
        $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 200<&- 2>&1 < /dev/null &

        mypid=$!

        # Add to pid file if successful start
        if [[ ${mypid} =~ ${IS_NUMBER} ]] && kill -0 $mypid > /dev/null 2>&1 ; then
            echo $mypid >> "$pid"
        else
            echo "Error starting $DAEMON daemon."
            exit 1
        fi
    ;;

    (stop)
        if [ -f "$pid" ]; then
            # Remove last in pid file
            to_stop=$(tail -n 1 "$pid")

            if [ -z $to_stop ]; then
                rm "$pid" # If all stopped, clean up pid file
                echo "No $DAEMON daemon to stop on host $HOSTNAME."
            else
                sed \$d "$pid" > "$pid.tmp" # all but last line

                # If all stopped, clean up pid file
                [ $(wc -l < "$pid.tmp") -eq 0 ] && rm "$pid" "$pid.tmp" || mv "$pid.tmp" "$pid"

                if kill -0 $to_stop > /dev/null 2>&1; then
                    echo "Stopping $DAEMON daemon (pid: $to_stop) on host $HOSTNAME."
                    kill $to_stop
                else
                    echo "No $DAEMON daemon (pid: $to_stop) is running anymore on $HOSTNAME."
                fi
            fi
        else
            echo "No $DAEMON daemon to stop on host $HOSTNAME."
        fi
    ;;

    (stop-all)
        if [ -f "$pid" ]; then
            mv "$pid" "${pid}.tmp"

            while read to_stop; do
                if kill -0 $to_stop > /dev/null 2>&1; then
                    echo "Stopping $DAEMON daemon (pid: $to_stop) on host $HOSTNAME."
                    kill $to_stop
                else
                    echo "Skipping $DAEMON daemon (pid: $to_stop), because it is not running anymore on $HOSTNAME."
                fi
            done < "${pid}.tmp"
            rm "${pid}.tmp"
        fi
    ;;

    (*)
        echo "Unexpected argument '$STARTSTOP'. $USAGE."
        exit 1
    ;;

esac
  • 使用flink-daemon.sh启动的flink使用的logback配置文件是logback.xml

flink-console.sh

flink-release-1.7.1/flink-dist/src/main/flink-bin/bin/flink-console.sh

#!/usr/bin/env bash
################################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
################################################################################

# Start a Flink service as a console application. Must be stopped with Ctrl-C
# or with SIGTERM by kill or the controlling process.
USAGE="Usage: flink-console.sh (taskexecutor|zookeeper|historyserver|standalonesession|standalonejob) [args]"

SERVICE=$1
ARGS=("${@:2}") # get remaining arguments as array

bin=`dirname "$0"`
bin=`cd "$bin"; pwd`

. "$bin"/config.sh

case $SERVICE in
    (taskexecutor)
        CLASS_TO_RUN=org.apache.flink.runtime.taskexecutor.TaskManagerRunner
    ;;

    (historyserver)
        CLASS_TO_RUN=org.apache.flink.runtime.webmonitor.history.HistoryServer
    ;;

    (zookeeper)
        CLASS_TO_RUN=org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer
    ;;

    (standalonesession)
        CLASS_TO_RUN=org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint
    ;;

    (standalonejob)
        CLASS_TO_RUN=org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint
    ;;

    (*)
        echo "Unknown service '${SERVICE}'. $USAGE."
        exit 1
    ;;
esac

FLINK_TM_CLASSPATH=`constructFlinkClassPath`

log_setting=("-Dlog4j.configuration=file:${FLINK_CONF_DIR}/log4j-console.properties" "-Dlogback.configurationFile=file:${FLINK_CONF_DIR}/logback-console.xml")

JAVA_VERSION=$(${JAVA_RUN} -version 2>&1 | sed 's/.*version "\(.*\)\.\(.*\)\..*"/\1\2/; 1q')

# Only set JVM 8 arguments if we have correctly extracted the version
if [[ ${JAVA_VERSION} =~ ${IS_NUMBER} ]]; then
    if [ "$JAVA_VERSION" -lt 18 ]; then
        JVM_ARGS="$JVM_ARGS -XX:MaxPermSize=256m"
    fi
fi

echo "Starting $SERVICE as a console application on host $HOSTNAME."
exec $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" ${CLASS_TO_RUN} "${ARGS[@]}"
  • 使用flink-console.sh启动的flink使用的logback配置文件是logback-console.xml

yarn-session.sh

flink-release-1.7.1/flink-dist/src/main/flink-bin/yarn-bin/yarn-session.sh

#!/usr/bin/env bash
################################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
################################################################################

bin=`dirname "$0"`
bin=`cd "$bin"; pwd`

# get Flink config
. "$bin"/config.sh

if [ "$FLINK_IDENT_STRING" = "" ]; then
        FLINK_IDENT_STRING="$USER"
fi

JVM_ARGS="$JVM_ARGS -Xmx512m"

CC_CLASSPATH=`manglePathList $(constructFlinkClassPath):$INTERNAL_HADOOP_CLASSPATHS`

log=$FLINK_LOG_DIR/flink-$FLINK_IDENT_STRING-yarn-session-$HOSTNAME.log
log_setting="-Dlog.file="$log" -Dlog4j.configuration=file:"$FLINK_CONF_DIR"/log4j-yarn-session.properties -Dlogback.configurationFile=file:"$FLINK_CONF_DIR"/logback-yarn.xml"

export FLINK_CONF_DIR

$JAVA_RUN $JVM_ARGS -classpath "$CC_CLASSPATH" $log_setting org.apache.flink.yarn.cli.FlinkYarnSessionCli -j "$FLINK_LIB_DIR"/flink-dist*.jar "$@"
  • 使用yarn-session.sh启动的flink使用的logback配置文件是logback-yarn.xml

小结

  • client端使用logback的话，要在pom文件添加logback-core、logback-classic及log4j-over-slf4j依赖，之后对flink-java、flink-streaming-java_2.11、flink-clients_2.11等配置log4j及slf4j-log4j12的exclusions；最后通过mvn dependency:tree查看是否还有log4j12，以确认下是否都全部排除了
  • 服务端使用logback的话，要在添加logback-classic.jar、logback-core.jar、log4j-over-slf4j.jar到flink的lib目录下(比如/opt/flink/lib)；移除flink的lib目录下(比如/opt/flink/lib)log4j及slf4j-log4j12的jar(比如log4j-1.2.17.jar及slf4j-log4j12-1.7.15.jar)；如果要自定义logback的配置的话，可以覆盖flink的conf目录下的logback.xml、logback-console.xml或者logback-yarn.xml
  • 使用flink-daemon.sh启动的flink使用的logback配置文件是logback.xml；使用flink-console.sh启动的flink使用的logback配置文件是logback-console.xml；使用yarn-session.sh启动的flink使用的logback配置文件是logback-yarn.xml

doc

logback logback配置 log4j配置 log4j

zhangyou00

zhangyou00

0 关注 0 粉丝 0 动态

关注 关注

相关推荐

学习logback配置的验证学到了哪些东西？

logback-classic：它是log4j的一个改良版本，同时它完整实现了slf4j API使你可以很方便地更换成其它日志系统如log4j或JDK14 Logging

JavaWinner 2020-03-04

使用springboot默认的日志系统logback打印Mybatis语句问题

如果需要保存打印的SQL到文件就不能设置为StdOutImpl，可以设置为Slf4jImpl，也可以不设置。然后对应接口所在包设置logback对应包的日志等级

zhangxiaocc 2020-02-23

【Log】【Logback】【1】简单介绍

--定义日志文件的存储地址 勿在 LogBack 的配置中使用相对路径-->. --格式化输出：%d表示日期，%thread表示线程名，%-5level：级别从左显示5个字符宽度%msg：日志消息，%n是换行符-->

丨Fanny丨Cri 2020-02-23

日志框架-Logback

1）一套能实现日志输出的工具包。2）能够描述系统运行状态的所有时间都可以算作日志。2）定制输出格式，可以通过配置文件修改输出信息。3）携带上下文信息。3）jboss-logging -面向的不是大众。5）Log4j2 -apache套用了log4j的名字，

丨Fanny丨Cri 2020-02-01

logback替换成log4j

logback和log4j区别和优势,简而言之就是logback更轻,更快,性能更好.具体的可以上官网查看.mybatis版本是3.2.8,框架是ssm.　　原来项目中使用的是log4j+slf4j,现在替换成logback_slf4j,所以可能会产生冲突

sdaq 2020-01-10

Java日志之Slf4j，Log4J，Logback原理总结

几乎任何应用，一定是需要日志的。那么，面对种类繁多的日志框架和配置，我们该何去何从?根本原因我也不太清楚，不过我猜测可能有这么一些理由，这些框架比较老，当初还没有slf4j这种事实上的标准，另一方面，有一些特殊的定制化的日志。严格意义上说slf4j自身并不

wangruiling 2020-01-07

logback介绍和配置详解

logback是java的日志开源组件，是log4j创始人写的，性能比log4j要好，目前主要分为3个模块。logback-classic:log4j的一个改良版本，同时实现了slf4j的接口，这样你如果之后要切换其他日志组件也是一件很容易的事。本篇博客会

sdaq 2019-12-28

logback模板

程序中记录日志一般有两个目的，第一个目的是做故障定位，第二个目的是显示程序运行状态。好的日志记录方式可以提供给我们足够多的定位问题的依据。我们这里引用Logback组件进行日志的记录。- logback-access这个模块是与service容器集成，提供

JavaWinner 2019-12-09

&lt;转&gt;LogBack简易教程

　　LogBack是一个日志框架，它与Log4j可以说是同出一源，都出自Ceki Gülcü之手。　　Slf4j是The Simple Logging Facade for Java的简称，是一个简单日志门面抽象框架，它本身只提供了日志Facade API

MrLiar 2016-05-15

SpringBoot优雅地配置日志

本文主要给大家介绍SpringBoot中如何通过sl4j日志组件优雅地记录日志。其实，我们入门 JAVA 的第一行代码就是一行日志，那你现在还在使用System.out.println("Hello,小明!Log4j作为当时作为最先比较流行的日志

SailingAptech 2019-09-05

logback记日志真的很好使

Logback is intended as a successor to the popular log4j project, picking up where log4j leaves off.logback扩展了log4j很多功能，以前一直不明白lo

狼烟烽火 2016-01-23

&lt;转&gt;LogBack简易教程

　　LogBack是一个日志框架，它与Log4j可以说是同出一源，都出自Ceki Gülcü之手。　　Slf4j是The Simple Logging Facade for Java的简称，是一个简单日志门面抽象框架，它本身只提供了日志Facade API

wanmeishijie 2016-05-15

springboot实现基于logback的日志配置

前言java web 下有好几种日志框架，比如：logback，log4j，log4j2。其中性能最高的应该使 logback 了，而且 springboot 默认使用的也是 logback 日志，所以本篇将会详细的讲解 logback 的日志配置方案。默

lzz0 2019-07-08

Spring Boot 之 LogBack 配置

默认情况下 Spring Boot 是以 INFO 级别输出到控制台。配置文件可命名为 logback.xml ， LogBack 自动会在 classpath 的根目录下搜索配置文件，不过 Spring Boot 建议命名为 logback-spring

LeeEmmmm 2019-07-01

MDC入门

具体的内容在有一个页面：。简单来说就是日志的增强功能，如果配置了MDC，并添加了相应的keyvalue，就会在打日志的时候把key对应的value打印出来。内部是用ThreadLocal来实现的，可以携带当前线程的context信息。MDC中包装了一个MD

sdaq 2016-08-04

logback的简单使用

logback当前分成三个模块：logback-core,logback- classic和logback-access。logback-core：其它两个模块的基础模块。此外logback-classic完整实现SLF4J API使你可以很方便地更换成其

SailingAptech 2019-06-30

【spring boot2】第3篇：spring boot中的日志框架

slf4j的使用我们知道 slf4j 是一个日志门面，具体的日志实现是由具体的日志框架实现的，比如 log4j、logback等日子框架。如下图所示，slf4j集成日志框架需要导入的 jar 包:当我们使用某个日志框架时，只需要在我们的系统类路径下配置对应

zhangyayun0 2019-06-30

storm 打印debug日志

storm采用了slf4j+logback方式，storm根路径下有个logback文件夹，下面有cluster.xml，对storm日志的配置。想要输出debug模式的数据，修改Supervisors下cluster.xml文件即可，

warmor 2015-12-14

SLF4J+logBack 代替 Log4j

作为日志领域中的新秀，已经有种颠覆领域的趋势了。SLF4J会根据classpath中所存在的适配器的jar来决定将使用的日志实现库。需要在程序开始时调用SLF4JBridgeHandler.install()来注册listener。拼装消息被推迟到了它能够

SEG的命名空间 2015-02-11

日志组件logback的介绍及配置使用方法(一)

Logback是由log4j创始人设计的又一个开源日志组件。logback当前分成三个模块：logback-core,logback- classic和logback-access。logback-core是其它两个模块的基础模块。logback-clas

由零开始 2015-02-11
zhangyou00

zhangyou00

W3CSchool教程
HTML 教程
CSS 教程
Bootstrap 教程
Javascript 教程
jQuery 教程
后端教程
C 教程
Java 教程
PHP 教程
Python 教程
Go 教程
移动开发
Android 教程
Swift 教程
Kotlin 教程
jQuery Mobile 教程
ionic 教程
关于我们
新闻动态
联系方式
招聘英才
安科实验室
帮助与反馈

安科网(Ancii)，中国第一极客网

Copyright © 2013 - 2019 Ancii.com

京ICP备18063983号 京公网安备11010802014868号