FilteredStream with query #426

CDeas9 · 2022-11-04T17:24:44Z

CDeas9
Nov 4, 2022

I am currently trying to switch from https://github.com/twitter/hbc and I'm having some trouble doing everything that could be done in there. The main 3 things are searching with a query, keeping the stream open and adding tweets to a queue as it searches. I think I have two of the three with this code.

this.clientV2 = new TwitterClient(TwitterCredentials.builder()
				.accessToken(this.token)
				.accessTokenSecret(this.tokenSecret)
				.apiKey(this.consumerKey)
				.apiSecretKey(this.consumerSecret)
				.build());

IAPIEventListener listener = new IAPIEventListenerImpl();
Future<Response> response = this.clientV2.startFilteredStream(listener);

In the IAPIEventListenerImpl I have

    public void onTweetStreamed(Tweet tweet) {
        try {
            ObjectWriter ow = new ObjectMapper().writer();
            String json = ow.writeValueAsString(tweet);
            StreamHandler.tweetQueue.add(json);
        } catch (JsonProcessingException e) {
            LOG.error(e.getMessage(), e);
        }
    }

to catch the tweets. The one thing I cant figure out is how to use a query. I see you can in searchAllTweets, but that doesnt stay open. Is there a way to combine those somehow?

This is the code I am trying to replace if that's helpful.

		Authentication auth = new OAuth1(
				this.consumerKey,
				this.consumerSecret,
				this.token,
				this.tokenSecret);
		Hosts hosts = new HttpHosts(Constants.STREAM_HOST);
		StatusesFilterEndpoint endpoint = new StatusesFilterEndpoint();

		List<String> terms = StreamHandlerConstantModule.getReducedQuery();
		endpoint.trackTerms(terms);

		ClientBuilder clientBuilder = new ClientBuilder()
				.hosts(hosts)
				.authentication(auth)
				.endpoint(endpoint)
				.processor(new StringDelimitedProcessor(tweetQueue));
		this.twitterClient = clientBuilder.build();

redouane59 · 2022-11-05T09:05:26Z

redouane59
Nov 5, 2022
Maintainer

Hey @CDeas9 , adding the filter rules and opening the stream are two different methods.
Here is a sample of my twitter bot for the stream part only if it can help :

public void start() {

    try {
      System.out.println("start streaming... ");
      this.streamResponse = TWITTER_CLIENT.startFilteredStream(new IAPIEventListener() {
        @Override
        public void onStreamError(final int httpCode, final String error) {
          System.err.println("HttpCode : " + httpCode + " - error : " + error);
          boolean result = TWITTER_CLIENT.stopFilteredStream(streamResponse);
          System.err.println("stopping stream... " + result);
          Thread.sleep(1000L * 5 * 60);
          System.err.println("restarting after 5min sleep");
          start();
        }

        @Override
        public void onTweetStreamed(final Tweet tweet) {
          try {
            Optional<TweetContent> tweetContent = incomingTweetsManager.prepareAnswer(tweet);
            if (tweetContent.isPresent()) {
              TweetContent content = tweetContent.get();
              TweetParameters tweetParameters = TweetParameters.builder()
                                                               .text(content.getText())
                                                               .reply(Reply.builder().inReplyToTweetId(content.getInReplyToStatusId()).build())
                                                               .build();
              if (content.getMediaId() != null) {
                tweetParameters.setMedia(Media.builder().mediaIds(List.of(content.getMediaId())).build());
              }
              TWITTER_CLIENT.postTweet(tweetParameters);
            }
          } catch (Exception e) {
            System.err.println("error streaming tweet : " + e.getMessage());
            System.err.println(Arrays.toString(e.getStackTrace()));
          }
        }

        @Override
        public void onUnknownDataStreamed(final String json) {
          System.err.println("onUnknownDataStreamed " + json);
          boolean result = TWITTER_CLIENT.stopFilteredStream(streamResponse);
          System.err.println("stopping stream... " + result);
        }

        @Override
        public void onStreamEnded(final Exception e) {
          System.err.println("onStreamEnded " + e.getMessage() + " / " + e.getStackTrace()[0]);
          boolean result = TWITTER_CLIENT.stopFilteredStream(streamResponse);
          System.err.println("stopping stream... " + result);
        }
      });
    } catch (Exception e) {
      System.err.println("- error - " + e.getMessage());
      if (streamResponse != null) {
        boolean result = TWITTER_CLIENT.stopFilteredStream(streamResponse);
        System.err.println("catch stopping stream... " + result);
      }
    }
  }

To add rules, you can check addFilteredStreamRule methods.
Queries are more used to search for tweets in oneshot, not in streaming.

11 replies

CDeas9 Dec 8, 2022
Author

Hello again, I am finally starting to test and I am having trouble connecting. Should the access tokens and everything for api 1.0 still work with TwitterClient?

redouane59 Dec 9, 2022
Maintainer

Yes this part didn't change. What is the problem exactly ?

CDeas9 Dec 13, 2022
Author

Its throwing the "NoSuchElementException" at

twittered/src/main/java/io/github/redouane59/twitter/helpers/RequestHelperV2.java

Line 145 in b8bf00f

    
           getTwitterCredentials().setBearerToken(result.orElseThrow(NoSuchElementException::new).getAccessToken());

which makes me think the makeRequest is failing. The code calling it is this.clientV2.addFilteredStreamRule(StreamHandlerConstantModule.getReducedQueryString(), "");

CDeas9 Dec 13, 2022
Author

This is the whole error
ERROR 2022-12-13 09:36:15.505 AbstractRequestHelper(111) Error occupied on executing request javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:369) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:312) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:199) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1488) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1394) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:441) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:412) at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1375) at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1350) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:220) at com.github.scribejava.core.httpclient.jdk.JDKHttpClient.prepareConnectionForBodyAndGetOutputStream(JDKHttpClient.java:232) at com.github.scribejava.core.httpclient.jdk.JDKHttpClient.addBody(JDKHttpClient.java:200) at com.github.scribejava.core.httpclient.jdk.JDKHttpClient.access$100(JDKHttpClient.java:25) at com.github.scribejava.core.httpclient.jdk.JDKHttpClient$BodyType$1.setBody(JDKHttpClient.java:152) at com.github.scribejava.core.httpclient.jdk.JDKHttpClient.doExecute(JDKHttpClient.java:134) at com.github.scribejava.core.httpclient.jdk.JDKHttpClient.execute(JDKHttpClient.java:94) at com.github.scribejava.core.oauth.OAuthService.execute(OAuthService.java:114) at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:85) at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:76) at io.github.redouane59.twitter.helpers.RequestHelperV2.getBearerToken(RequestHelperV2.java:144) at io.github.redouane59.twitter.helpers.RequestHelperV2.signRequest(RequestHelperV2.java:131) at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:83) at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:76) at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:54) at io.github.redouane59.twitter.helpers.RequestHelperV2.postRequest(RequestHelperV2.java:118) at io.github.redouane59.twitter.TwitterClient.addFilteredStreamRule(TwitterClient.java:1203) at org.battelle.horizon.stream.StreamHandler.setupClient(StreamHandler.java:249) at org.battelle.horizon.stream.StreamHandler.<init>(StreamHandler.java:215) at org.battelle.horizon.stream.StreamHandler.<init>(StreamHandler.java:169) at org.battelle.horizon.HorizonScanning.pipeline(HorizonScanning.java:51) at org.battelle.horizon.HorizonScanning.main(HorizonScanning.java:222) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at java.base/sun.security.validator.Validator.validate(Validator.java:264) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ... 37 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ... 42 more

but sometimes I get
HttpCode : 403 - error : {"client_id":"14029529","detail":"When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal.","registration_url":"https://developer.twitter.com/en/docs/projects/overview","title":"Client Forbidden","required_enrollment":"Standard Basic","reason":"client-not-enrolled","type":"https://api.twitter.com/2/problems/client-forbidden"}

CDeas9 Dec 19, 2022
Author

I think the problem is that the account we have now only has an app and it isnt associated to a project. The account isnt accessible, so we have to sign up for a new one.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FilteredStream with query #426

{{title}}

Replies: 1 comment 11 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

FilteredStream with query #426

CDeas9 Nov 4, 2022

Replies: 1 comment · 11 replies

redouane59 Nov 5, 2022 Maintainer

CDeas9 Dec 8, 2022 Author

redouane59 Dec 9, 2022 Maintainer

CDeas9 Dec 13, 2022 Author

CDeas9 Dec 13, 2022 Author

CDeas9 Dec 19, 2022 Author

CDeas9
Nov 4, 2022

Replies: 1 comment 11 replies

redouane59
Nov 5, 2022
Maintainer

CDeas9 Dec 8, 2022
Author

redouane59 Dec 9, 2022
Maintainer

CDeas9 Dec 13, 2022
Author

CDeas9 Dec 13, 2022
Author

CDeas9 Dec 19, 2022
Author