-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error otx-misp import with the latest MISP #37
Comments
Dang sorry to see this. I have the same issue. I also found -d works and with -vv it shows the OTX side is working but not the commit and -vv does not show anything more when -d is removed. Name: pymisp |
switched to using cabby to pull from OTX and then use stix-to-misp.py to upload. |
Hi, I'm going to test this method. I don't know cabby, so it's going to be time to test the tool. Can Cabby export in stix format? Thanks for your idea |
It does stix by default. Here is what we are doing for this part.
We are also looking at contributing back to misp-stix-converter an option to add tags to the uploaded items. |
Hi, I have always the same error: 2020-03-04 20:25:03,098 INFO: Polling using data binding: ALL If I do this command: Nota: I have add the verbose option. I have this logs: 2020-03-04 20:48:51,191 cabby.cli.poll INFO: Polling using data binding: ALL 2020-03-04 20:48:51,195 urllib3.connectionpool DEBUG: Starting new HTTPS connection (1): otx.alienvault.com:443 2020-03-04 20:48:54,449 cabby.client11.Client11 INFO: 3 services discovered 2020-03-04 20:48:54,452 urllib3.connectionpool DEBUG: Starting new HTTPS connection (1): otx.alienvault.com:443 2020-03-04 20:48:57,459 cabby.cli.commons ERROR: FAILURE: There was a failure while executing the message handler If you have any idea .... |
When using "taxii-poll --path https://otx.alienvault.com/taxii/poll -c user_AlienVault --username MY_API_KEY --dest-dir /tmp" the user_AlienVault does not have any alerts in it. Did you subscribe to someone and try to poll their feed? Also, the --host and --path were frustrating for me as well. |
Yes, I have subscribe to the AlienVault user I have change "taxii-poll --path https://otx.alienvault.com/taxii/poll -c AlienVault --username MY_API_KEY --dest-dir /tmp" But i have the same error |
something is weird with that one. I just subscribed to user jnazario and I get the same error but go back to Metadefender and get a 504 then ran it again and got a 502 then again and it worked (bottom). It might be a busy server or gremlins in the interwebs or something. taxii-collections --path https://otx.alienvault.com/taxii/collections --username MY_KEY --password foo
|
I have subscribe to the user popularmalware and it's work for me... perhaps a problem with the Alienvault user .. |
Good to hear it worked for you. Maybe we need to take the debug to Alienvault. |
Hi, I work on the OTX team. I'll look into this as soon as I can and get back to you. |
hi, |
I don't know too much about otx-misp but it's probably the same problem that occurs with Cabby, which is that there's an error encountered when trying to render the AlienVault user's feed to STIX. |
OK, so, I believe that I have fixed the problem related to not being able to get TAXII results for the collection "user_AlienVault" (and possibly some other users had the sample problem, the error was related to converting some types of data to STIX and only affected users who had examples of that kind of data) If that doesn't fix the otx-misp problems let me know and I can look deeper into it. As an aside - if you're using Cabby for anything other than testing I would recommend limiting the results by date. The AlienVault user has a LOT of pulses/indicators, and not providing a date means that it downloads all of them, which is quite slow/taxing. Providing a date will only get pulses that are new or have been modified since that date, so if you know you fetched N hours ago, then limiting to that time frame will greatly reduce fetch time and traffic. Such as Ideally you'd use --begin with whatever your last fetch date was, and just keeping feeding "new" indicators into whatever you're using. |
yes it works :) |
Thanks J-Robinson042519 for misp-stix-converter I have changed "/usr/local/lib/python3.6/dist-packages/misp_stix_converter/converters/convert.py" I added: Thanks a lot J-Robinson042519 and rustybrooks |
@V1D1AN - did you commit this to the converter project? If not please do or make it a config parameter and add it to the project. I don't think we will get to it for a while. Does anyone know how to limit the polling on otx-misp like @rustybrooks did for taxii-poll? |
looks like with the fix from @rustybrooks did not help otx-misp...still getting Error: 'TypeError' object has no attribute 'message'. One theory we have is it is coming from MISP with its Py 2.7 conversation issues. |
@V1D1AN - Where in the code did you add the misp_event section? around like 210 looks about right. |
Good morning, MISP version: 2.4 If I add -d, it works perfectly. |
May you help us? As I'm seeing I'm not the only one with this issue. Thanks! |
I've actually been playing with otx-misp today, and I have the same problem as you. I think what it comes down to is that otx-misp relies on another library called pymisp. The most recent version of pymisp has a PyMISP() object that does not take the same init parameters as it used to, so otx-misp fails. I've tried downgrading to an older version of pymisp, and that gets past this problem, but there are other problems down the road. pymisp is handling the MISP api calls and my guess is that how these calls works has changed. If I update to a newer pymisp and fix the initialization problems then I run into the fact that pymisp uses different functions - the "new_event" function no longer exists for example. So I think it's fixable but possibly not trivial to do so. I'm still looking into it, and other alternatives such as offering a MISP feed directly from OTX |
Note, a fork of otx-misp has fixed this problem I believe: I myself did not go down this road. Here's the problem as I see it
I kind of gave up on fixing this and I'm going to instead, I think, make MISP feeds for OTX. I did the first one here, which lets you get a MISP feed from a group, as a POC I will probably add more MISP feed support in the near future Note: edited to fix URL |
Hi, How am I supposed to get this feed working? Is the URL really GROUPID? How about source format in MISP? Is it MISP Feed then? For what I tested it didn't work with this information. Thank you. |
GROUPID should be the numeric group id of the group you want to sync. For example this is a newly created public group: It's a MISP feed, i.e. it has a /manifest.json and then a json file per pulse in the feed. This url would be to sync a single group. We'll be adding the ability to sync a user and probably also your subscription |
Hi, On OTX Side, I created a group with 2 pulses. Any idea? Thanks a lot ! |
In that URL you need to replace GROUPID with the integer group id for your group. For example if you go to a group in the UI, the url will look something like And of course YOURKEYHERE in the headers would have to be your OTX api key. If you have access to the MISP logs I think you'll find errors in there. I'm not currently running a MISP instance but I could maybe fire one up again if necessary to try to check it out. |
Thanks @rustybrooks . I had, of course, replaced all the values to be changed. ;-) Here are logs:
|
Aw man sorry I had a typo in the URL, it should be |
👌 Perfect! It works like a charm! Many thanks for this workaround @rustybrooks! I hope we will have soon a new version of otx-misp. |
Thanks for this @rustybrooks . It is very nice to have OTX pulses as MISP events. But I guess I found something to report you: In MISP there are "email-dst" and "email-src" attributes. As I understood in OTX only one attribute exist which is "email". When I checked OTX events in my MISP they are always "email-dst". Can you fix it? I believe OTX "email" should map to MISP "email-src". |
Yeah, I can change that, sure. It probably does make more sense for it to be email-src which I guess is probably the "from" address |
Thanks @rustybrooks for this well needed solution. I am quite new to using OTX, but have seen that many authors create new pulses every day, rather than updating an existing one. Unless I'm mistaken, this means that me adding that pulse to a Group is good for that day and no more. Have you managed to explore putting similar MISP manifests on either Users, or just the subscribed pulses? I have had no end of difficulties with trying to get OTX-MISP working which I acknowledge is probably mainly user error/ignorance, this solution would be invaluable to me! |
Yeah there are plans to add MISP support to users, and/or making a MISP feed for your "subscription" which includes:
Might come through before the end of this year but is not immediately planned |
The error 'TypeError' object has no attribute 'message' is happening again...it started very recently for me. In fact, I was receiving an error message stating that I had to move to the latest version of MISP to continue, then after I did, I started getting these TypeError messages. I've been trying to find a way around this and ended up here. AFAIK, we were pulling all public pulses in the primary Alienvault feed (I believe that means I'm pulling all public pulses from user Alienvault). There doesn't appear to be a group number for that main public feed. If there is, please let me know as I would love to try getting the feeds using this other method. We are using a config file, but running it from the manually from the CLI with the verbose switch provided no additional information. Any help would be appreciated. |
@tonycaud have you tried the fork from this project I mentioned above? |
Since I simply installed using "pip3 install otx-misp" how do I see which I have? |
Then you have this one, almost surely. I think you'd have to download the fork from github and install it manually. |
So, that one throws a chunk of errors, and there isn't an issues page there... -nevermind...with help from a friend of mine, we changed it from calling Python2.7 to Python3 and now it seems to be working. |
Description
Versions
Traceback or error
When i execute:
otx-misp -o xxxxxxxxxxxxxxxxxx -s https://127.0.0.1 -m xxxxxxxxxxxxxxxxxxxxxxxx
I have always the error :
Error: 'TypeError' object has no attribute 'message'
The text was updated successfully, but these errors were encountered: