Skip to content
This repository has been archived by the owner on Feb 19, 2024. It is now read-only.

Validator restarts #275

Open
feeleep75 opened this issue May 12, 2021 · 1 comment
Open

Validator restarts #275

feeleep75 opened this issue May 12, 2021 · 1 comment

Comments

@feeleep75
Copy link

feeleep75 commented May 12, 2021 •
edited
Loading

Hi - I am getting this kind of issues recently:

# 2021-05-12T03:56:21,606 [DEBUG/BFTSync/ConsensusRunner vb1q26cz] (BFTSync.java:443) - SYNC_VERTICES: Received GetVerticesErrorResponse: GetVerticesErrorResponse{vb1q0x47->HighQC[highest=QC{37:53223}, highestCommitted=<same>, highestTC=null]} highQC: HighQC[highest=QC{37:53223}, highestCommitted=<same>, highestTC=null]
java.lang.NullPointerException
	at com.radixdlt.consensus.sync.BFTSync.processGetVerticesLocalTimeout(BFTSync.java:318)
	at com.radixdlt.consensus.epoch.EpochManager.processGetVerticesLocalTimeout(EpochManager.java:408)
	at io.reactivex.rxjava3.internal.observers.LambdaObserver.onNext(LambdaObserver.java:63)
	at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.drainNormal(ObservableObserveOn.java:201)
	at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.run(ObservableObserveOn.java:255)
	at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:322)
	at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker.runEager(ExecutorScheduler.java:287)
	at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:248)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
# 2021-05-12T03:56:21,607 [ERROR/EpochManagerRunner/ConsensusRunner vb1q26cz] (EpochManagerRunner.java:213) - Unexpected exception occurred
java.lang.NullPointerException: null
	at com.radixdlt.consensus.sync.BFTSync.processGetVerticesLocalTimeout(BFTSync.java:318) ~[radixdlt-1.0-beta.32.jar:?]
	at com.radixdlt.consensus.epoch.EpochManager.processGetVerticesLocalTimeout(EpochManager.java:408) ~[radixdlt-1.0-beta.32.jar:?]
	at io.reactivex.rxjava3.internal.observers.LambdaObserver.onNext(LambdaObserver.java:63) ~[rxjava-3.0.9.jar:?]
	at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.drainNormal(ObservableObserveOn.java:201) ~[rxjava-3.0.9.jar:?]
	at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.run(ObservableObserveOn.java:255) ~[rxjava-3.0.9.jar:?]
	at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:322) ~[rxjava-3.0.9.jar:?]
	at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker.runEager(ExecutorScheduler.java:287) ~[rxjava-3.0.9.jar:?]
	at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:248) ~[rxjava-3.0.9.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
+ find /usr -type f -name java -exec setfattr -n user.pax.flags -v em {} ;
+ env
+ sort
CORE_GOSSIP_PORT=30000
HOME=/root
HOSTNAME=853478c1ba60
JAVA_OPTS=-server -Xms4g -Xmx4g -XX:+HeapDumpOnOutOfMemoryError -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts -Djavax.net.ssl.trustStoreType=jks -Djava.security.egd=file:/dev/urandom -Dcom.sun.management.jmxremote.port=9010 -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=core -agentlib:jdwp=transport=dt_socket,address=50505,suspend=n,server=y

# 2021-05-12T03:56:24,541 [INFO/MemoryLeakDetector/main] (MemoryLeakDetector.java:54) - Starting memory leak detector...
2021-05-12 03:56:24,548 Log4j2-TF-3-AsyncLoggerConfig-2 ERROR An exception occurred processing Appender FILE java.lang.NullPointerException
	at org.apache.logging.log4j.util.StackLocator.calcLocation(StackLocator.java:79)
	at org.apache.logging.log4j.util.StackLocatorUtil.calcLocation(StackLocatorUtil.java:78)
	at org.apache.logging.log4j.core.impl.Log4jLogEvent.getSource(Log4jLogEvent.java:693)
	at org.apache.logging.log4j.core.pattern.FileLocationPatternConverter.format(FileLocationPatternConverter.java:57)
	at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
	at org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:345)
	at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:244)
	at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:229)
	at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:59)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:197)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:190)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:181)
	at org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender.append(RollingRandomAccessFileAppender.java:252)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
	at org.apache.logging.log4j.core.async.AsyncLoggerConfig.callAppenders(AsyncLoggerConfig.java:127)
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
	at org.apache.logging.log4j.core.async.AsyncLoggerConfig.log(AsyncLoggerConfig.java:121)
	at org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncLoggerConfigsOnCurrentThread(AsyncLoggerConfig.java:169)
	at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:111)
	at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:97)
	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
	at java.base/java.lang.Thread.run(Thread.java:834)

2021-05-12 03:56:24,554 Log4j2-TF-3-AsyncLoggerConfig-2 ERROR An exception occurred processing Appender STDOUT java.lang.NullPointerException
	at org.apache.logging.log4j.util.StackLocator.calcLocation(StackLocator.java:79)
	at org.apache.logging.log4j.util.StackLocatorUtil.calcLocation(StackLocatorUtil.java:78)
	at org.apache.logging.log4j.core.impl.Log4jLogEvent.getSource(Log4jLogEvent.java:693)
	at org.apache.logging.log4j.core.pattern.FileLocationPatternConverter.format(FileLocationPatternConverter.java:57)
	at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
	at org.apache.logging.log4j.core.pattern.HighlightConverter.format(HighlightConverter.java:245)
	at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
	at org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:345)
	at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:244)
	at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:229)
	at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:59)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:197)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:190)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:181)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
	at org.apache.logging.log4j.core.async.AsyncLoggerConfig.callAppenders(AsyncLoggerConfig.java:127)
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
	at org.apache.logging.log4j.core.async.AsyncLoggerConfig.log(AsyncLoggerConfig.java:121)
	at org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncLoggerConfigsOnCurrentThread(AsyncLoggerConfig.java:169)
	at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:111)
	at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:97)
	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
	at java.base/java.lang.Thread.run(Thread.java:834)

# 2021-05-12T03:56:24,641 [INFO/PersistedProperties/main] (PersistedProperties.java:61) - Loaded properties from default.config
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/radixdlt/lib/guice-4.2.3.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
@cknowles
Copy link

cknowles commented Aug 7, 2021

Was getting the same error for the log4j file appender part in the current 1.0.0 Docker setup. The node still appears to carry on and works, however I ended up customising the log4j configuration to get rid of it. Turned off the file appender at the same time since I didn't want it writing a file inside the container (not good practice).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cknowles @feeleep75