聊聊jdk httpclient的retry参数

序

本文主要研究一下jdk httpclient的retry参数

DEFAULT_MAX_ATTEMPTS

java.net.http/jdk/internal/net/http/MultiExchange.java

class MultiExchange<T> {

    static final Logger debug =
            Utils.getDebugLogger("MultiExchange"::toString, Utils.DEBUG);

    private final HttpRequest userRequest; // the user request
    private final HttpRequestImpl request; // a copy of the user request
    final AccessControlContext acc;
    final HttpClientImpl client;
    final HttpResponse.BodyHandler<T> responseHandler;
    final HttpClientImpl.DelegatingExecutor executor;
    final AtomicInteger attempts = new AtomicInteger();
    HttpRequestImpl currentreq; // used for retries & redirect
    HttpRequestImpl previousreq; // used for retries & redirect
    Exchange<T> exchange; // the current exchange
    Exchange<T> previous;
    volatile Throwable retryCause;
    volatile boolean expiredOnce;
    volatile HttpResponse<T> response = null;

    // Maximum number of times a request will be retried/redirected
    // for any reason

    static final int DEFAULT_MAX_ATTEMPTS = 5;
    static final int max_attempts = Utils.getIntegerNetProperty(
            "jdk.httpclient.redirects.retrylimit", DEFAULT_MAX_ATTEMPTS
    );

    //......

}

• 这里有一个AtomicInteger类型的attempts变量，用来记录请求次数
• 另外还有一个max_attempts，读取的是jdk.httpclient.redirects.retrylimit值，读取不到默认取DEFAULT_MAX_ATTEMPTS，为5

MultiExchange.responseAsyncImpl

java.net.http/jdk/internal/net/http/MultiExchange.java

private CompletableFuture<Response> responseAsyncImpl() {
        CompletableFuture<Response> cf;
        if (attempts.incrementAndGet() > max_attempts) {
            cf = failedFuture(new IOException("Too many retries", retryCause));
        } else {
            if (currentreq.timeout().isPresent()) {
                responseTimerEvent = ResponseTimerEvent.of(this);
                client.registerTimer(responseTimerEvent);
            }
            try {
                // 1. apply request filters
                // if currentreq == previousreq the filters have already
                // been applied once. Applying them a second time might
                // cause some headers values to be added twice: for
                // instance, the same cookie might be added again.
                if (currentreq != previousreq) {
                    requestFilters(currentreq);
                }
            } catch (IOException e) {
                return failedFuture(e);
            }
            Exchange<T> exch = getExchange();
            // 2. get response
            cf = exch.responseAsync()
                     .thenCompose((Response response) -> {
                        HttpRequestImpl newrequest;
                        try {
                            // 3. apply response filters
                            newrequest = responseFilters(response);
                        } catch (IOException e) {
                            return failedFuture(e);
                        }
                        // 4. check filter result and repeat or continue
                        if (newrequest == null) {
                            if (attempts.get() > 1) {
                                Log.logError("Succeeded on attempt: " + attempts);
                            }
                            return completedFuture(response);
                        } else {
                            this.response =
                                new HttpResponseImpl<>(currentreq, response, this.response, null, exch);
                            Exchange<T> oldExch = exch;
                            return exch.ignoreBody().handle((r,t) -> {
                                previousreq = currentreq;
                                currentreq = newrequest;
                                expiredOnce = false;
                                setExchange(new Exchange<>(currentreq, this, acc));
                                return responseAsyncImpl();
                            }).thenCompose(Function.identity());
                        } })
                     .handle((response, ex) -> {
                        // 5. handle errors and cancel any timer set
                        cancelTimer();
                        if (ex == null) {
                            assert response != null;
                            return completedFuture(response);
                        }
                        // all exceptions thrown are handled here
                        CompletableFuture<Response> errorCF = getExceptionalCF(ex);
                        if (errorCF == null) {
                            return responseAsyncImpl();
                        } else {
                            return errorCF;
                        } })
                     .thenCompose(Function.identity());
        }
        return cf;
    }

• 进入该方法的时候，调用attempts.incrementAndGet()，递增请求次数，然后判断有无超出限制，有则返回带有new IOException("Too many retries", retryCause)异常的failedFuture，即通过CompletableFuture.completeExceptionally返回
• 如果没有超出限制，但是执行请求失败，则调用getExceptionalCF来判断是否应该重试，如果返回null，则重试，通过再次调用responseAsyncImpl，通过这种递归调用完成重试逻辑

MultiExchange.getExceptionalCF

java.net.http/jdk/internal/net/http/MultiExchange.java

/**
     * Takes a Throwable and returns a suitable CompletableFuture that is
     * completed exceptionally, or null.
     */
    private CompletableFuture<Response> getExceptionalCF(Throwable t) {
        if ((t instanceof CompletionException) || (t instanceof ExecutionException)) {
            if (t.getCause() != null) {
                t = t.getCause();
            }
        }
        if (cancelled && t instanceof IOException) {
            if (!(t instanceof HttpTimeoutException)) {
                t = toTimeoutException((IOException)t);
            }
        } else if (retryOnFailure(t)) {
            Throwable cause = retryCause(t);

            if (!(t instanceof ConnectException)) {
                if (!canRetryRequest(currentreq)) {
                    return failedFuture(cause); // fails with original cause
                }
            }

            // allow the retry mechanism to do its work
            retryCause = cause;
            if (!expiredOnce) {
                if (debug.on())
                    debug.log(t.getClass().getSimpleName() + " (async): retrying...", t);
                expiredOnce = true;
                // The connection was abruptly closed.
                // We return null to retry the same request a second time.
                // The request filters have already been applied to the
                // currentreq, so we set previousreq = currentreq to
                // prevent them from being applied again.
                previousreq = currentreq;
                return null;
            } else {
                if (debug.on()) {
                    debug.log(t.getClass().getSimpleName()
                            + " (async): already retried once.", t);
                }
                t = cause;
            }
        }
        return failedFuture(t);
    }

    private boolean retryOnFailure(Throwable t) {
        return t instanceof ConnectionExpiredException
                || (RETRY_CONNECT && (t instanceof ConnectException));
    }

    /** Returns true if the given request can be automatically retried. */
    private static boolean canRetryRequest(HttpRequest request) {
        if (RETRY_ALWAYS)
            return true;
        if (isIdempotentRequest(request))
            return true;
        return false;
    }

    /** Returns true is given request has an idempotent method. */
    private static boolean isIdempotentRequest(HttpRequest request) {
        String method = request.method();
        switch (method) {
            case "GET" :
            case "HEAD" :
                return true;
            default :
                return false;
        }
    }

    private Throwable retryCause(Throwable t) {
        Throwable cause = t instanceof ConnectionExpiredException ? t.getCause() : t;
        return cause == null ? t : cause;
    }

    /** True if ALL ( even non-idempotent ) requests can be automatic retried. */
    private static final boolean RETRY_ALWAYS = retryPostValue();
    /** True if ConnectException should cause a retry. Enabled by default */
    private static final boolean RETRY_CONNECT = retryConnect();

    private static boolean retryPostValue() {
        String s = Utils.getNetProperty("jdk.httpclient.enableAllMethodRetry");
        if (s == null)
            return false;
        return s.isEmpty() ? true : Boolean.parseBoolean(s);
    }

    private static boolean retryConnect() {
        String s = Utils.getNetProperty("jdk.httpclient.disableRetryConnect");
        if (s == null)
            return false;
        return s.isEmpty() ? true : Boolean.parseBoolean(s);
    }

如果cancelled为true且是IOException则直接返回，否则先判断retryOnFailure再判断canRetryRequest(如果不是ConnectException才走canRetryRequest这个判断)

• retryOnFailure方法判断如果是ConnectionExpiredException或者是ConnectException且开启retryConnect，则返回true
• RETRY_CONNECT读取的是jdk.httpclient.disableRetryConnect参数，如果值为null，则方法返回false，即不进行retryConnect
• canRetryRequest首先判断RETRY_ALWAYS，在判断isIdempotentRequest(GET、HEAD方法才重试)，都不是则返回false
• RETRY_ALWAYS读取的是jdk.httpclient.enableAllMethodRetry，如果值为null，则方法返回false，即不进行retryPostValue
• 如果该重试的话，则返回null，responseAsyncImpl里头在getExceptionalCF返回null的时候，重新调用了一次responseAsyncImpl，通过递归调用来完成重试逻辑

NetProperties

java.base/sun/net/NetProperties.java

public class NetProperties {
    private static Properties props = new Properties();
    static {
        AccessController.doPrivileged(
            new PrivilegedAction<Void>() {
                public Void run() {
                    loadDefaultProperties();
                    return null;
                }});
    }

    private NetProperties() { };


    /*
     * Loads the default networking system properties
     * the file is in jre/lib/net.properties
     */
    private static void loadDefaultProperties() {
        String fname = StaticProperty.javaHome();
        if (fname == null) {
            throw new Error("Can't find java.home ??");
        }
        try {
            File f = new File(fname, "conf");
            f = new File(f, "net.properties");
            fname = f.getCanonicalPath();
            InputStream in = new FileInputStream(fname);
            BufferedInputStream bin = new BufferedInputStream(in);
            props.load(bin);
            bin.close();
        } catch (Exception e) {
            // Do nothing. We couldn't find or access the file
            // so we won't have default properties...
        }
    }

    /**
     * Get a networking system property. If no system property was defined
     * returns the default value, if it exists, otherwise returns
     * <code>null</code>.
     * @param      key  the property name.
     * @throws  SecurityException  if a security manager exists and its
     *          <code>checkPropertiesAccess</code> method doesn't allow access
     *          to the system properties.
     * @return the <code>String</code> value for the property,
     *         or <code>null</code>
     */
    public static String get(String key) {
        String def = props.getProperty(key);
        try {
            return System.getProperty(key, def);
        } catch (IllegalArgumentException e) {
        } catch (NullPointerException e) {
        }
        return null;
    }

    /**
     * Get an Integer networking system property. If no system property was
     * defined returns the default value, if it exists, otherwise returns
     * <code>null</code>.
     * @param   key     the property name.
     * @param   defval  the default value to use if the property is not found
     * @throws  SecurityException  if a security manager exists and its
     *          <code>checkPropertiesAccess</code> method doesn't allow access
     *          to the system properties.
     * @return the <code>Integer</code> value for the property,
     *         or <code>null</code>
     */
    public static Integer getInteger(String key, int defval) {
        String val = null;

        try {
            val = System.getProperty(key, props.getProperty(key));
        } catch (IllegalArgumentException e) {
        } catch (NullPointerException e) {
        }

        if (val != null) {
            try {
                return Integer.decode(val);
            } catch (NumberFormatException ex) {
            }
        }
        return defval;
    }

    /**
     * Get a Boolean networking system property. If no system property was
     * defined returns the default value, if it exists, otherwise returns
     * <code>null</code>.
     * @param   key     the property name.
     * @throws  SecurityException  if a security manager exists and its
     *          <code>checkPropertiesAccess</code> method doesn't allow access
     *          to the system properties.
     * @return the <code>Boolean</code> value for the property,
     *         or <code>null</code>
     */
    public static Boolean getBoolean(String key) {
        String val = null;

        try {
            val = System.getProperty(key, props.getProperty(key));
        } catch (IllegalArgumentException e) {
        } catch (NullPointerException e) {
        }

        if (val != null) {
            try {
                return Boolean.valueOf(val);
            } catch (NumberFormatException ex) {
            }
        }
        return null;
    }

}

• 这里通过loadDefaultProperties先加载默认配置，读取的是JAVA_HOME/conf/net.properties文件
• 然后getString、getInteger、getBoolean方法采用的是System.getProperty来读取，而net.properties值仅仅作为System.getProperty的defaultValue
• 因此要设置httpclient相关参数，只需要通过System.setProperty或者-D来设置即可
• net.properties

/Library/java/JavaVirtualMachines/jdk-11.jdk/Contents/home/conf/net.properties

java.net.useSystemProxies=false
http.nonProxyHosts=localhost|127.*|[::1]
ftp.nonProxyHosts=localhost|127.*|[::1]
jdk.http.auth.tunneling.disabledSchemes=Basic

net.properties文件默认设置了如上四个参数

相关异常

HttpTimeoutException

java.net.http/java/net/http/HttpTimeoutException.java

/**
 * Thrown when a response is not received within a specified time period.
 *
 * @since 11
 */
public class HttpTimeoutException extends IOException {

    private static final long serialVersionUID = 981344271622632951L;

    /**
     * Constructs an {@code HttpTimeoutException} with the given detail message.
     *
     * @param message
     *        The detail message; can be {@code null}
     */
    public HttpTimeoutException(String message) {
        super(message);
    }
}

• 属于java.net.http包，继承至IOException
• 如果设置了request的timeout，则注册ResponseTimerEvent，在超时时抛出HttpTimeoutException: request timed out，同时设置MultiExchange的cancelled为true
• 这类由于客户端设置超时引起的HttpTimeoutException，不会进行重试，即使开启相关重试参数
• 如果这个时间设置得太短，则在connect的时候就超时了，这个时候会抛出HttpConnectTimeoutException，而非HttpTimeoutException: request timed out

HttpConnectTimeoutException

java.net.http/java/net/http/HttpConnectTimeoutException.java

/**
 * Thrown when a connection, over which an {@code HttpRequest} is intended to be
 * sent, is not successfully established within a specified time period.
 *
 * @since 11
 */
public class HttpConnectTimeoutException extends HttpTimeoutException {

    private static final long serialVersionUID = 321L + 11L;

    /**
     * Constructs an {@code HttpConnectTimeoutException} with the given detail
     * message.
     *
     * @param message
     *        The detail message; can be {@code null}
     */
    public HttpConnectTimeoutException(String message) {
        super(message);
    }
}

• 属于java.net.http包，继承至HttpTimeoutException
• 如果设置了client的connectTimeout，则会注册ConnectTimerEvent，在超时时抛出ConnectException("HTTP connect timed out")，同时设置MultiExchange的cancelled为true，这个在MultiExchange.getExceptionalCF方法里头会被包装为HttpConnectTimeoutException

ConnectionExpiredException

java.net.http/jdk/internal/net/http/common/ConnectionExpiredException.java

/**
 * Signals that an end of file or end of stream has been reached
 * unexpectedly before any protocol specific data has been received.
 */
public final class ConnectionExpiredException extends IOException {
    private static final long serialVersionUID = 0;

    /**
     * Constructs a {@code ConnectionExpiredException} with a detail message of
     * "subscription is finished" and the given cause.
     *
     * @param   cause the throwable cause
     */
    public ConnectionExpiredException(Throwable cause) {
        super("subscription is finished", cause);
    }
}

• 一般是在read error的时候触发，比如EOFException，IOException("connection reset by peer)，或者SSLHandshakeException

小结

jdk httpclient的retry参数涉及到的参数如下：

• jdk.httpclient.redirects.retrylimit(默认为5，用来控制重试次数，不过实际上还有expiredOnce参数，看代码貌似顶多重试一次)
• jdk.httpclient.disableRetryConnect(默认为null，即RETRY_CONNECT为false，不在ConnectException的时候retry)
• jdk.httpclient.enableAllMethodRetry(默认为null，即RETRY_ALWAYS为false，即需要判断请求方法是否幂等来决定是否重试)

是否重试的判断逻辑如下：

• 如果重试次数超过限制，则返回失败，否则往下
• 如果cancelled为true(这里如果request设置了timeout，触发时cancelled设置为true)且是IOException(例如设置了连接超时抛出的HttpConnectTimeoutException)，则不走重试逻辑；否则往下

• 如果retryOnFailure(ConnectionExpiredException，或者ConnectException且开启retryConnect)，则往下

  • 如果是异常不是ConnectException，则还额外判断canRetryRequest(判断该请求类型是否允许重试)，满足则往下
  • 如果expiredOnce为false，则返回null，即满足重试条件，走递归重试

doc

• HttpClient javadoc