Replies: 7 comments 11 replies
-
Thanks a lot that you are willing to use our tool. Just a quick conclusion, you can not project from raw to DOM directly due to the dimension loss
But you can achieve this in an alternative way (we are drafting this application for a journal now). You can first split the DOM into several grids, and project each grid to the corresponding region on the raw image (probably the closest one is enough). Then you can "intersect" them with your raw image detection results (or simply do detection on the projected region area). And then for each grid, calculate its own transformation matrix from DOM coordinate to raw image coordinate. Lastly for this grid, you can move the detection results on the raw image back to the DOM by the inverse transformation matrix. (Apologies we haven't had time to integrate this approach to an easy-to-use API in the EasyIDP before publishing the paper) Currently, limited by the heavy workload from other projects in the lab, the EasyIDP is not completely coded (too many complaints from co-workers for v1.0, we are completely reconstructing the API in v2.0 for easier use, coding almost 70% with docs only around 20%). For example, this batch processing code has not been implemented in the current v2.0.0.dev3: proj = idp.ProjectPool()
proj.add_metashape(["/Users/benweinstein/Dropbox/Weecology/everglades_species/easyidp/Hidden_Little_03_24_2022.psx"]) But alternatively, the base object for each metashape and pix4d project have been finished, you can try it equally by: ms = idp.Metashape("xxxx.psx", chunk_id=0) # please check the folder in xxxx.files.
ms.back2raw(roi) # save_folder parameters in the readme also not implemented yet. Then for the metashape project files in your dropbox, the And for the errored GeoTIFF file, very appreciate if you can share the problem GeoTIFF via Dropbox, it will be a good source for debugging, and I will try my best to find a solution for it. I guess it may have some important offset value missing or renamed in the 'XxxxCitationGeoKey', Also, just in case it is the problem caused by DOM exporting, here is the parameter settings that we used in the Metashape, provided here for your reference. For our agricultural proposes, using EPSG:4326 (lon, lat) may not a proper for size estimation. We prefer the UTM geo-coordinate instead because the units of x and y are in meters. (For Florida, it may be UTM zone 17N?) I believe you are more expert in this area than I am, please point out if anything I misunderstood about it. I have just update the Quick Start examples in the docs, please refer to it to check the main workflow by our dataset. (Please press Ctrl+Shift+R to refresh if you have a local browser cache.) For your applications, especially the grid spliting and inverse transformation matrix from raw image back to the DOM, I will find a time in this week to finish a similar demo for you based on our dataset. |
Beta Was this translation helpful? Give feedback.
-
Let's keep this conversation, clearly alot is in active development. Let's start with the code. All files are in the dropbox link above, include the .files, and the .tiff. I cloned your main branch and pulled.
Looks like we are missing the z dimension. I have downloaded agisoft and I am familiarizing myself with it to export new files and align with the expectations of the API. Probably a better error message is required above. I remade the 3D model in a very quick and coarse way, but I still get the error above. I think this relates to our larger conceptual mismatch that I think is worth discussing. I completely agree that you cannot recover a 2D position from a 3D point. However, I'm wondering about just multiplying the orthomosaic by the inverse transformation matrix for each camera position in 2D space. Clearly it will not be exact, but it may be roughly accurate to localize the bounding box. Happy to do any legwork/testing, we have hundreds of .psx projects from UAV flights. Here is that attached paper |
Beta Was this translation helpful? Give feedback.
-
In this weekend, I have tried my proposed "inverse transform matrix" workflow for your cases, you can access the jupyter notebook from here: https://drive.google.com/drive/folders/1VbQGwt2pWuoXUhx8i2tLBq18uLl8sDh6?usp=sharing Firstly, I export the DEM and also make a grid ( Then I using the "136" grid as example, and do the backward projection to the closest image, and manually add 2 detection results as test data: The agricultural cases are almost flat plane, when it comes to your cases (dramatic changes in the small area), seems get the problem you are concerned: According to this pre-experiment, I come up with the following solution:
The solution 2 is almost the similar idea like the paper you shared (get the z for raw pixels). But in that paper, it does not consider the lens distortion effects. If we want to take it into consideration, we have to export the undistorted images by Pix4D or Metashape. Otherwise, we have to solve the inverse equation for lens distortion (personally feel quite challenging for it): |
Beta Was this translation helpful? Give feedback.
-
Great, i've reviewed this. I think this is really promising, even with the offset observed in the orange box. I think we should consider this always as a product for machine learning. I think it is reasonable to assume in most cases that the artifacts in the orthomosaic inhibit classification much more than detection. So rather than coming up with a perfect photogrammetry solution, we can also use the machine learning model to run on the ortho to match to the closest detection. This of course will not be perfect in cluttered scenes, but I think it will be highly valuable none the less. Given that we have a reasonable way forward. Let me summarize to make sure I understand with some psuedo-code.
we'll also want to formalize those plotting functions. I can generate predictions for our object detection models in the raw and orthomosaic and then score the accuracy of the model. This would ask whether the potential increase in classification accuracy in the raw image is worth the potential drop in accuracy in the detection step. Long term i'm happy to talk about wrapping this into a manuscript, or I can just use it in our downstream work. My lab at University of Florida (https://www.weecology.org/) is focused on open data science for ecological monitoring. We are highly collaborative, whatever works for you. I'm happy to work on a pull request and you can comment, or you can lead. I can supply more data and do the evaluations. Lots of options, obviously everyone is busy, always. It remains absolutely remarkable how common I see this question among users of our object detection package and a number of threads on the agisoft forums (e.g https://www.agisoft.com/forum/index.php?topic=9404.0 or https://www.agisoft.com/forum/index.php?topic=2484.0 and many others), but that there isn't a more well defined solution/workflow. Strangely enough Agisoft has picked up our deepforest tool and encoded it in a GUI on their scripts page (https://agisoft.freshdesk.com/support/solutions/articles/31000162552-automatic-detection-of-objects-on-orthomosaic). I will be messaging the developer as we get further along. |
Beta Was this translation helpful? Give feedback.
-
I was very excited to learn about the EasyIDP project. I'm happy to see this progress toward projecting from raw images -> orthomosaics. I would find this extremely useful as well! |
Beta Was this translation helpful? Give feedback.
-
I'm starting to spend more time on this project over the next few weeks. Nice new docs. I think some of the upgrades have made new assumptions about the file structure. Using the same data from above (#44 (comment)). you now get
which you can see when debugging on L1178 in metashape.py is because the sensor dictionary is indexed 0
but the camera.sensor_id is 1
My gut reaction is that sensors is a dict with keys
but camera.sensor_id is an int. So that lookup won't work. This was after I renamed the np.float -> float, which i'm going to make a PR for now. Reminder the data is linked: #44 (comment) |
Beta Was this translation helpful? Give feedback.
-
Yes, fixed. Running tutorial now. Thanks!
…On Fri, Mar 31, 2023 at 7:53 PM Howcanoe WANG ***@***.***> wrote:
9d4929c
<9d4929c>
should fix this
—
Reply to this email directly, view it on GitHub
<#44 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJHBLECSIQPNTJNGYAMLUDW66KCBANCNFSM6AAAAAAQEEPM3U>
.
You are receiving this because you authored the thread.Message ID:
<UTokyo-FieldPhenomics-Lab/EasyIDP/repo-discussions/44/comments/5494810@
github.com>
--
Ben Weinstein, Ph.D.
Research Scientist
University of Florida
http://benweinstein.weebly.com/
|
Beta Was this translation helpful? Give feedback.
-
Summary
Can I use EasyIDP to find the location in the orthomosaic from a point in the raw images? This is the opposite workflow of the current project. This workflow would be widely applicable to machine learning object detection pipelines and be of interest to a large audience. I am happy to contribute to a PR if this fits the scope of this project. It is very possible that this functionality already lives inside the project, but is not seen as a primary user workflow (see below).
Use Case
I am a machine learning researcher at the University of Florida (https://scholar.google.com/citations?user=7POnELAAAAAJ&hl=en) I develop python tools and datasets for object detection in airborne imagery https://deepforest.readthedocs.io/.
I am often asked whether it is better to predict in the raw images versus the orthomosaics created from UAV imagery. Reading the paper (https://www.mdpi.com/2072-4292/13/13/2622), we are in agreement of the importance of using the highest quality imagery.
For example, we have a nice 6 species bird detection model we have developed for our UAV orthomosaics.
but we often see that these georeferenced orthomosaics can have significant artifacts, likely because the trees/birds move among images, leading very blurred pixels. However, in wildlife ecology, we often want to use a georefenced image for time-series, or counting, or other use cases. Therefore if we make predictions in the raw images, we still need to find their location in the orthomosaic. FYI, the orthos were made before they get to me, so i'm still learning the agisoft terminology, but I have a solid remote sensing background.
Current State
The package currently provides functionality to identify the location in the raw images from selecting an ROI in the orthomosaic. Under this setup, a current object detection workflow would be
if we had a raw to orthomosaic mapper the workflow would be
Existing solutions
I see a class that I think forms the basis of this work:
EasyIDP/easyidp/reconstruct.py
Line 401 in 0d074ca
EasyIDP/easyidp/metashape.py
Line 93 in 0d074ca
but they are not documented and look like _internal functions.
What I've tried so far.
Here is the package of data I'm working from.
https://www.dropbox.com/sh/1gc5k0kvryhnxnm/AAANNgZHl05SrIfHnt1mz8Kca?dl=0
I made a sample ROI
A couple errors that I ran into
first, the proj object is empty
When trying read the ortho, I get
but it reads fine in rasterio
This could easily be due to my lack of familiarity with agisoft, I doubt it implies a larger issue. Once we get these sorted out we can discuss the larger questions above.
All thoughts welcome, thanks for your work on this important and overlooked area of development.
Beta Was this translation helpful? Give feedback.
All reactions