Please open source this library #2
Comments
|
Open sourcing the upper MAC layer is certainly on our roadmap. This does require a fair amount of developer time though, because large codebase has to be refactored, cleaned up, and documented. At this point we see more benefit in spending developer resources on those many features missing from the ESP-IDF. As the ESP-IDF gets more feature complete, we will invest more time into MAC layer clean up and open sourcing. |
|
Well that's great news! |
|
If you release what you have now, we'll figure it out. We can even help you out! The point of having an open codebase is to have a constructive, bidirectional relationship with your users instead of releasing software for mere consumption. |
|
The problem mostly is that what we have now isn't designed to be released and has developed some ball-of-yarn qualities. We need time to separate the bits of the MAC that we can open-source from the lower layers that need to stay closed because they contain IP-related sensitive information. As you can see, well, everywhere else in esp-idf, we're absolutely not against having a constructive and bidirectional relationship with our users; it's one of the reasons you see a gazillion commits to the esp-idf master branch and not a closed release full of binaries every x time! We just ask you to be patient on this one, we know it's a highly requested feature but due to the nature of the thing, we can't just throw it out in the open, sorry. |
|
What would it take to open source the entire thing? Could we crowdfund however much it would take to buy the IP-protected parts free? How much would that cost and with whom would we negotiate? |
|
I can't say... I'll pass it on, see if that is an option. |
|
Sorry, just got a response; unfortunately in this case throwing money at the problem isn't going to help. |
|
Dear espressif I am now at same thinking. i do not want further more time wait for fixing things in the lib. please open the libs that we can develope on Open Source way. best wishes |
|
@igrr Sorry to bother you here, but can we get some status update on upper layer open sourcing process? This is essential for a production-grade systems built with ESP32. Thanks in advance. |
|
I am hearing rumors saying Espressif will never open source this library. That's a shame, especially because the company is just riding the open source wave to spread within the community. All I can say, they only care about selling hardware along with their proprietary, closed-source, bad quality software. All in all, great marketing play. Well done! |
|
Folks, indeed the process is a long one, but we are taking steps in opening parts of these libraries. Since this issue was opened, significant portion of rtc library was released, and recently ets_timer implementation was opened as well. In the foreseeable future, some parts (such as the PHY library) are going to stay closed source. But the supplicant and eventually upper MAC layer will be opened. Thank you for your patience and understanding. |
|
Does this apply to the Bluetooth part as well? Since BLE shares pretty much the same Phy as NRF24L01 it would be interesting to bypass the protocol stack to ease interoperability. |
|
Hi @igrr. |
|
Any updates from espressif? I would also love for the wifi stack to be open source. |
|
@stern0m1 no specific updates at the moment, but some parts of the supplicant (WPA2 and WPS) are planned to be open sourced in release 3.1. |
|
Thanks. |
|
No, esp-now is built upon vendor IE, so it doesn't let you receive any extra packets. |
|
If esp-now wouldn't filter by vendor I would be able to receive a command from an iPhone with one click. Direct without establishing a network. This is another example of the benefits of open sourcing... I have spent many hours/days trying to figure out a way to receive a command on the esp8266 from an iPhone with one click. Any ideas? |
|
Is it possible to open things up enough to route the timings of the wlan-ap master beacons out in order to do some time synching on them? That would be awesome and a big feature. (Multi Room Audio, Spatially separated Microphone Arrays...) |
|
Any update on the open source release process ? Is this still in the roadmap of the 3.1 at least for the Wifi Part ? |
|
Parts of WPA2 and WPS source code are planned to be open sourced in 3.1. |
|
@igrr thank you for everything may we know when will mac layer library will be open sourced? |
|
Recently WPA2 and WPS code has been moved to IDF. Next we plan to move more of wpa_supplicant code into IDF, but the exact scope for the next step is not determined yet. I will update this issue once more concrete plans are in place. |
|
@igrr sure we will wait for your update on this issue now? thanks |
|
After years of struggling with dodgy WiFi drivers, the embedded systems community finally has a vendor who is committed to delivering transparency and quality in their WiFi software! The open-sourcing that has already occurred (particularly in wpa_supplicant) made a huge difference in the confidence that product designers can put in ESP32. Keep it up! |
|
Yes, open source this, please. I've just gotten 4 of Gosund's 220V switches, it's using this lib internally. I don't trust it on my WLAN without locking it out and putting it in a VLAN. Source needs to be audited. |
Actually, writing a single line of code is the easiest in many cases. You can't do it with a broomstick, so any alternative is most likely not easier, nor necessarily targeted, nor the point. The point I was trying to make is that when you're already making iot projects, having a very simple way to fully control the wifi antenna, will definately lead to alot more annoyance/interference in the functionality of any nearby devices, whereas without that access, then there is far less likelihood that people will have the motivation or the capability (inadvertent or otherwise), to actually affect nearby communications beyond hogging bandwidth by doing alot of streaming.
that is not easier, nor as cheap as flashing an app that intentionally or unintentionally ends up jamming communications elsewhere, the likelihood of which would increase if given easy access to the lowest wifi layer. Anyways, I agree with the Non-Disclosure opinions... that's also mostly likely. But then espressif is at a disadvantage if all the alternatives are allowed to disclose it, and they are, as for the nRF I hear it's wifi stack is fully documented at the register access level. |
What is easier? Open door of microwawe oven and press button? Even if you are right and jamming is hard, communication is still harder, so in the world without open wifi docs jammer is even have higher chance to be implemented. After all nothing stops from going to shop and buying jammer for about 2-3 euros which is even cheaper than esp+programmer or 8-12 euros to buy jammer for 5GHz, 2.4GHz, 1.8GHz, 800 MHz and 433 trash heap. So flashing some evil-evil firmware is not the easiest nor the cheapest way of jamming. |
Wifi SoC with fully documented registers I know is Atheros' one. |
|
Hi @igrr |
|
Alongside WiFi, their mesh code is also closed-sourced. Making it very hard to find problems. It is time we start to reverse-engineer everything ESP related. |
|
@devrim-oguz you can try using my reverse engineering (patching) framework: https://github.com/Matheus-Garbelini/esp32_firmware_patching_framework You can explore symbols of a compiled firmware of a ble mesh program with ghidra (including ROM space) and then use a python script which updates the compiled ELF file to hook custom callback functions to addresses of your choice or substitute some instructions to others. |
|
jack will say no. |
|
in 2023 again :) please open source this library we want start now - legal - with build linux driver for the esp32-s3 thank you |
|
local ping @igrr want ping www now . - )
|
Does it work with rf phy directly through registers? |
... this try happens on this path asap next time - |
|
Open sourcing this library (or at least the MAC part) would also enable the community to add support for 802.11 extensions, for example 802.11s (mesh networking on the MAC layer). |
|
Will libwapi.so support wapi-cert auth mode? if not, can you opensource it or give some interface let us implement it? |
|
If the goat does not come to the milking shed, the milking shed will come to the goat! We started reverse engineering the ESP32 802.11 MAC and PHY layer, see https://zeus.ugent.be/blog/23-24/open-source-esp32-wifi-mac/. The open-source implementation can currently connect to an open access point and send UDP packets. The goal is to have a completely open-source implementation of the PHY and MAC layer: at the moment, we still do the radio initialisation using the Espressif libraries. After the initialisation is done, we kill the proprietary wifi FreeRTOS task, replace the interrupt with our own interrupt and only run open-source code from then on. As for legalities, we're in the clear, since the blobs are licensed under the Apache 2.0 license, see https://opensource.stackexchange.com/a/9568 I invite you all to read the blog post, and then maybe collaborate on reverse engineering or re-implementing wifi functionality. |
|
+1
I really want to implement FTM-Broadcast outlined by this paper. Unfortunately the ftm session code is hidden behind the API call. |
|
Progress report: we are now able to connect to an open AP, and use the regular ESP32 networking API, so you can now write for example a HTTP server (previously, the UDP packets were hardcoded). We're currently working on radio initialization to be able to entirely remove the binary blobs; this is going smoothly. See also https://esp32-open-mac.be/ for progress reports, blogs posts and a post where I built a cheap (~300 EUR) Faraday cage with data passthrough. All code is in https://github.com/esp32-open-mac/. |
Wow, great job. This is really impressive. |
You guys are awesome for doing this |
|
I will be giving a talk at the Gulaschprogrammiernacht 22 conference in Germany (https://cfp.gulas.ch/gpn22/talk/TZXJBW/) about this. Feel free to come if you're interested! |
|
+1. I'm aware, that Espressif likely can't open source the lower parts of the stack for IP reasons. I also get the argument of preventing people from making deauthers. It was however extremely trivial to just patch out the check preventing the transmission of any frames. I'm also pretty sure, that Espressif could open source the upper parts of the stack, since they are evidently ported from FreeBSD. Nordic's WiFi stack is open source as well, except for the lowest layers. |
|
I will be giving a talk about this at the RIOT Summit 2024 in Austria (https://summit.riot-os.org/2024/blog/speakers/jasper-devreker/). September 5th or 6th, free entrance. Feel free to come if you're interested! |
|
The recording from the talk is available: https://www.youtube.com/watch?v=sGdD8XWTM2Y |
|
in 2024 again and again. we run on each espressif factory problem on a big wall of silent. the factory firmware should run without problems on a new product. no indications that what, what does not work or is restricted. no note as an enclosure that 138240 baud are necessary because 48 MHz are installed. we are slowly but surely getting tired of the long search and trial and unanswered problem reports that cost a lot of time. sorry guys but now it's close to the end. I am sorry. i actually expect people to take it seriously and to heart when products are delivered also that the Factory FW is tested and goes only with QM over the product to sales! I am sorry again. when a new product is highly advertised with new standards ( Dual Wifi 5GHz ) and advertisements, the factory firmware should be put through its paces. why doesn't anyone dedicate themselves to this? what should we check and run on the ESP32-C5 with Dual WiFi 6 , the LED flashing works, great, but the WiFi 6 5GHz not as expected. there is silence, ducking away, persevering. i don't want to know what money this group devours, burns, who deliver such shameful factory firmware. every 5th grader can do that better if you make the sources available to him. my 5ct. and that's it. espressif should finally understand that it does more harm if you are constantly silent - than admitting mistakes. last but least : open the binary blobs and we can work byself on this problems. in the end, it frustrates. and you get goosebumps. i'm also slowly but surely running out of patience for something like that. then it should have been. then better an end with horror than a horror without end. @igrr and enter |
|
I would like to make it clear that ESP32DE is not affiliated with the esp32-open-mac project in any way. While the esp32-open-mac project would like some support from Espressif (mostly documentation or being able to ask the engineers some questions), we know we're not entitled to anything, and are already very grateful that the blobs are released under an open source license that allows reverse engineering. @ESP32DE these rude comments/rants don't contribute in any way towards building or releasing an open source Wi-Fi driver, and are likely even counterproductive: by acting rudely towards Espressif employees, you discourage work or support from them on this. Could you please remove the comment or move it to somewhere more relevant? |
heyja @redfast00 WoW 👍 switch on OT How do you come to the conclusion that I have something to do with your project and justify that I have nothing to do with your project in any way? I have been speaking for myself and my concerns since 2013 - not for you or others. And I confirm to you that I have nothing to do with your project in any way. This has been going on here since 2013 :) How do you come to the conclusion that this issue belongs to you alone and take the right to manage it? Do you really seriously believe that reverse engineering only started since your project announcement quote: "If the goat does not come to the milking shed, the milking shed will come to the goat" quote end in December 2023? And no, I'm not going to start a fight here and start a war of words. Just read my statement, it doesn't concern you personally and it's not directed at you personally, nor at your project, nor at anything else that could have anything to do with you as property, but at the binary blob that we've been discussing since August 2013 - where were you then? There were no espressifers, no repos or gits. There were just a handful of people who got together to form an LTD ... in Hong Kong and have been doing engineering ever since ... what is now called Espressif Systems - But all of this doesn't contribute to the matter, just like your statement and your accountability and now my statement to yours. The latter is solely yours personally to me directed post in public and will only happen once. I think I've told you everything I needed to say. I actually didn't want to write back, not just because it's just a "test of strength" but because it's completely off topic, doesn't contribute anything to the problem here and just bores people here. And no, I won't answer any questions for your project if you have any on this topic. See how you get on :) Can we get back to the topic now? Thank you very much! Have a nice day too. "One more direct address like that and you'll be blocked" no, you won't hear that from me - that's nonsense. But I'm just not going to respond to such postings anymore, I'm just going to keep quiet, which doesn't mean that I'll accept it and put up with it. You just rise above it. No arrogance, but calm in the box. switch off OT |
|
@ESP32DE Again, I apologize for this issue. |
|
Progress update: instead of also porting FreeBSDs 80211 stack (the way Espressif chose), we've started writing a new MAC stack in Rust. It can currently scan channels for beacon frames and connect to a pre-defined open network. There is an integration with ESP-NETIF, so once the ESP32 connects to the network, you can ping it/send HTTP requests/... We've also reverse engineered the 80211 RX filters (https://esp32-open-mac.be/posts/0008-rx-filter/), so now the ESP32 doesn't have to process all packets that are in the air just to drop 90% of them. |
|
That's amazing! Sounds like the open source process could render in a technically better solution then… ;-) |
|
Progress updates: we've implmented a SoftAP mode. There is now also a Wi-Fi stack completely written in Rust that already implements STA mode, including scanning (made by @Frostie314159). He also started implementing AWDL (Apple Wireless Direct Link). @mjwells2002 reverse engineered the Nintendo DSi PictoChat protocol and started using our library to implement a transmitter that can send pictures. I've started reverse engineering the Wi-Fi cryptography acceleration used in WPA and so far it works; there should be another blog post about it soon. @Frostie314159 and me are giving a talk about reversing the ESP32 Wi-Fi hardware at the Chaos Communication Congress (https://fahrplan.events.ccc.de/congress/2024/fahrplan/talk/C38ZK7/). They've given us a big room, so there appears to be a lot of interest in having an open source Wi-Fi stack. The talk will be recorded, but if you're there, come say hi :) If you want to meet up, you can contact us via the Matrix linked on https://esp32-open-mac.be/ |
and others
Having an open source WiFi MAC on such an affordable device would be extremely useful for experimenters and researchers.
If you are worried about people disregarding the WiFi standard then that is already possible with other devices.
The existing open WiFi drivers for cheap routers such as the AR9xxx driver that exists for the Omega2 $5 OpenWRT-compatible micro-router is easily configurable in ways that are harmful, outside of the WiFi specification, and illegal in various countries.
Software Defined Radios allow complete control over the WiFi MAC and PHY and these devices are getting cheaper every year.
Many devices already speak other non-WiFi protocols on the same 2.4 GHz frequencies as WiFi and standards-compliant WiFi devices knows how to handle the interference.
If you open source the WiFi MAC then the rest of the world can help improve your firmware to make it the best microcontroller WiFi MAC available.
Given all of this I do not understand why you would keep the WiFi MAC closed.
The text was updated successfully, but these errors were encountered: