Relocalize a smartphone on a preloaded point cloud - object-recognition

Being a novice I need an advice how to solve the following problem.
Say, with photogrammetry I have obtained a point cloud of the part of my room. Then I upload this point cloud to an android phone and I want it to track its camera pose relatively to this point cloud in real time.
As far as I know there can be problems with different cameras' (simple camera or another phone camera VS my phone camera) intrinsics that can affect the presision of localisation, right?
Actually, it's supposed to be an AR-app, so I've tried existing SDKs - vuforia, wikitude, placenote (haven't tried arcore yet cause my device highly likely won't support it). The problem is they all use their own clouds for their services and I don't want to depend on them. Ideally, it's my own PC where I perform 3d reconstruction and from where my phone downloads a point cloud.
Do I need a SLAM (with IMU fusion) or VIO on my phone, don't I? Are there any ready-to-go implementations within libs like ARtoolKit or, maybe, PCL? Will any existing SLAM catch up a map, reconstructed with other algorithms or should I use one and only SLAM for both mapping and localization?
So, the main question is how to do everything arcore and vuforia does without using third party servers. (I suspect the answer is to device the same underlay which vuforia and other SDKs use to employ all available hardware..)

Related

Relocalize a smartphone on a preloaded point cloud

Being a novice I need an advice how to solve the following problem. Say, with photogrammetry I have obtained a point cloud of the part of my room. Then I upload this point cloud to an android phone and I want it to track its camera pose relatively to this point cloud in real time.
As far as I know there can be problems with different cameras' (simple camera or another phone camera VS my phone camera) intrinsics that can affect the presision of localisation, right?
Actually, it's supposed to be an AR-app, so I've tried existing SDKs - vuforia, wikitude, placenote (haven't tried arcore yet cause my device highly likely won't support it). The problem is they all use their own clouds for their services and I don't want to depend on them. Ideally, it's my own PC where I perform 3d reconstruction and from where my phone downloads a point cloud.
Do I need a SLAM (with IMU fusion) or VIO on my phone, don't I? Are there any ready-to-go implementations within libs like ARtoolKit or, maybe, PCL? Will any existing SLAM catch up a map, reconstructed with other algorithms or should I use one and only SLAM for both mapping and localization?
So, the main question is how to do everything arcore and vuforia does without using third party servers. (I suspect the answer is to device the same underlay which vuforia and other SDKs use to employ all available hardware..)
Any help is highly appreciated!
Thank you in advance,
Sophour

The utility and use of ARCore with image recognition

If I understand correctly, the ARCore does not yet take image recognition into account. Correct me if I'm wrong but, a priori, each time the sdk is coupled with another framework (eg. vuforia) or a library (eg. opencv). These technologies already make it possible to superimpose information on an image, so I would like to know what functionality (s) is being developed with the ARCore in this kind of project, in what way is it relevant to use this sdk.
For example: the Google's Daydream Labs project - "how to make espresso"
(https://www.youtube.com/watch?v=Dz4nLqpmPjs)
When passing in front of certain components of the coffee machine, the device displays different information. There is no plane detection and tracking in this project. So what has been developed with the ARCore?
Thank you in advance for your answers
Just taking a guess here but that "how to make an espresso" project is probably doing object recognition using 3d spatial points. So in theory a different machine could potentially trigger the same tutorial even though it lacks the parts or are in slightly different locations.
I've been trying all sorts of image recognition plugins in conjunction with ARCore and non seem to really work. They pretty much fight with ARcore.
Augmented Images API is available for ARCore developers since ARCore 1.2 was released.
Augmented Images in ARCore lets you build AR apps that can respond to 2D images, such as posters or product packaging, in the user's environment. You provide a set of reference images, and ARCore tracking tells you where those images are physically located in an AR session, once they are detected in the camera view.

Fingerprint Scanner using Camera [closed]

Working on fingerprint scanner using camera or without, its possibility and its success rate?, I came across one of open source SDK named FingerJetFX its provide feasibilty with android too.
The FingerJetFX OSE fingerprint feature extractor is platform-independent and can be built
for, with appropriate changes to the make files, and run in environments with or without
operating systems, including
Linux
Android
Windows
Windows CE
various RTOSs
but I'm not sure whether Fingerprint scanner possible or not, I download the SDK and digging but no luck, even didn't found any steps to integrate the SDK, so having few of question which listed below:
I'm looking for suggestion and guidance:
Fingerprint scanner can be possible in android using camera or without camera?
With the help of FingerJetFX can I achieve my goal?
If 2nd answer is yes, then can someone provide me any sort of steps to integrate SDK in android?
Your suggestion are appreciable.
Android Camera Based Solutions:
As someone who's done significant research on this exact problem, I can tell you it's difficult to get a suitable image for templating (feature extraction) using a stock camera found on any current Android device. The main debilitating issue is achieving significant contrast between the finger's ridges and valleys. Commercial optical fingerprint scanners (which you are attempting to mimic) typically achieve the necessary contrast through frustrated total internal reflection in a prism.
In this case, light from the ridges contacting the prism are transmitted to the CMOS sensor while light from the valleys are not. You're simply not going to reliably get the same kind of results from an Android camera, but that doesn't mean you can't get something useable under ideal conditions.
I took the image on the left with a commercial optical fingerprint scanner (Futronics FS80) and the right with a normal camera (15MP Cannon DSLR). After cropping, inverting (to match the other scanner's convention), contrasting, etc the camera image, we got the following results.
The low contrast of the camera image is apparent.
But the software is able to accurately determine the ridge flow.
And we end up finding a decent number of matching minutia (marked with red circles.)
Here's the bad news. Taking these types of up close shots of the tip of a finger is difficult. I used a DSLR with a flash to achieve these results. Additionally most fingerprint matching algorithms are not scale invariant. So if the finger is farther away from the camera on a subsequent "scan", it may not match the original.
The software package I used for the visualizations is the excellent and BSD licensed SourceAFIS. No corporate "open source version"/ "paid version" shenanigans either although it's currently only ported to C# and Java (limited).
Non Camera Based Solutions:
For the frightening small number of devices that have hardware that support "USB Host Mode" you can write a custom driver to integrate a fingerprint scanner with Android. I'll be honest, for the two models I've done this for it was a huge pain. I accomplished it by using wireshark to sniff USB packets between the scanner and a linux box that had a working driver and then writing an Android driver based on the sniffed commands.
Cross Compiling FingerJetFX
Once you have worked out a solution for image acquisition (both potential solutions have their drawbacks) you can start to worry about getting FingerJetFX running on Android. First you'll use their SDK to write a self contained C++ program that takes an image and turns it into a template. After that you really have two options.
Compile it to a library and use JNI to interface with it.
Compile it to an executable and let your Android program call it as a subprocess.
For either you'll need the NDK. I've never used JNI so I'll defer to the wisdom of others on how best us it. I always tend to choose route #2. For this application I think it's appropriate since you're only really calling the native code to do one thing, template your image. Once you've got your native program running and cross compiled you can use the answer to this question to package it with your android app and call it from your Android code.
Tthere are a couple immediate hurdles:
Obtaining a good image of the fingerprint will be critical. According to their site, fingerjet expects standard fingerprint images - e.g. 8-bit greyscale (high contrast), flattened fingerprint images. If you took fingerprint pictures with the camera, the user would need to have a flat transparent surface (glass) you could flatten the fingerprints onto in order to take the picture. Your app would then locate the fingerprint in the image, transform it into a format acceptable for fingerjet. A library like OpenCV would help do this.
FingerJetFX OSE does not appear to offer canned android support - you will have to compile the library for android and use it via JNI/NDK.
From there, fingerjet should provide you with a compact representation of the print you can use for matching.
It would be feasible, but the usage requirement (need for the user to have a flat transparent surface available) might be a deal breaker...

How to identify 3d objects using android phone [closed]

i am developing an android app. i want my app to recognize a 3d object to which the camera of my phone is exposed. currently i am able to achieve 2d object identification. however i would like to know how can 3d object identification be made possible in android. for eg if a user of my app focuses the camera lens on a microwave, the app should recognize this microwave is of Panasonic.
I went throught Google Googles, however it is just able to identify 2d objects and my app can do that anyway. i would like to know a way to identify 3d objects.
This is because there is no publicly available library or API for object recognition that handles 3D image files.
Considering that there is no standard file format for 3D images at this time, this is to be expected. Most 3D image files are propriety, usually combining two 2D images in a standard format like JPEG in their own container. A better solution if you want the user experience in 3D is to show the user the 3D image and send the server the 2D counterpart (involving converting the 3D format on the device to 2D onboard or taking a second picture in close succession in 2D).
Bottom line is that this will not work with Google Goggles or OpenCV. Even those are nowhere near being real-time. Remember, we are using slow computers with slow internet connections and low-end cameras.
As an Evo3D owner myself, I can understand your want. The Technology is just ahead of the available software at this time.
I hope this information helps. I found out answer for the problem stated above. Aurasma is tool developed for Android and IOS that has the capability to identify real life objects. so referring to the same example as stated above, your android app containing aurasma kernel will identify the microwave and depending on what info you have supplied it will showcase that info every time it comes across the target image/product.
Even if Google has not come out with object identification technique, Aurasma has provided us with that solution. I think enrolling in PhD and doing my specialization in 3D machine vision was a longer process.. Some people showed poor maturity while replying to this post.. No issues.. if Aurasma kernel serves your purpose i will b happy..
Cheers..

How to Detect Physical object in Android Augmented Reality?

I found many ways to detect different shapes. But hard luck when I am going for a physical object. From what I read we should have a black border around images to make a pattern file. If I follow this concept and generate a pattern then my application detects images on printout. But in the real world a physical object not necessarily has a black border square shape around it.
Update
Although I accept an answer, my question remains unsolved. As there is still no solution for detecting physical object.
Any further research and links are welcome!
The bad news is, you can't use AndAR to detect physical objects. AndAR is based on a fiducial marker approach, where the marker is made of two components: a solid border and an interior pattern. The pattern encodes a value that can be used to address a particular model to render on the marker, and the border makes it easy to determine the relative orientation of the marker to the device. Clearly this is just planar image recognition.
To do object recognition on a 3D object is a more complicated problem, and I don't know of any Android libraries that provide a turn-key solution, but recognizing just one object is probably feasible on a mobile device.
One possibility might be to investigate the available Android AR toolkits (Layar, Junaio, Qualcomm AR SDK) which all now support some image recognition. It may be that by taking images of your teapot at various rotations and using those as the images you want your app to match against that you might get this solution working, but keep in mind they are only designed to do planar matching on images, not real 3D objects, so the performance might not be great. (Well, the Metaio Mobile SDK Pro does 3D recognition and tracking, but it's very expensive).
While object recognition is perhaps best done by comparing camera frames with images of the object you wish to recognize (or by comparing image features from the camera frames with pre-computed image features etc), tracking is a different matter. If you want to accurately track your 3D object in 3D space you'll really need to have or build a 3D model of it, and for each frame determine point correspondences between the camera image and the 3D object for tracking. True unassisted (i.e. no depth-camera) 3D tracking is hard.
I hope this gives you some background you can use to evaluate your next steps.
Update:
Qualcomm's Vuforia SDK allows you to track "multi targets", which are objects with a set of planar tracking surfaces with a fixed spatial relationship. If you made a "cube" different photos of the 6 sides of your object (teapot) that might work somewhat.
https://ar.qualcomm.at/qdevnet/developer_guide/Trackables
End of 2013 Update:
I have no experience with these, but:
Metaio now offer 3D tracking of CAD models: https://dev.metaio.com/sdk/tutorials/3d-tracking-based-on-cad-data/
ARLab.com's LinkAR promises object matching. http://www.arlab.com/objecttracking
I would note the use of the word "matching" - I think the use case here is you know the object you want to overlay (a toy-box, and engine etc). Differentiating between multiple 3D objects may be entirely out of scope.
I recently read about research being doing on hierarchical shape vocabularies used for object representation. Of course, there is no library available for download, but in case you are interested in the general approach here you can find some papers.
Also you might be interested in this paper. It describes an algorithm for detecting objects based on a set of contours.
Have you considered OpenCV for detecting and tracking object? Take a look at Find Objects with a Webcam tutorial (C++ / Qt). OpenCV is available for Android - OpenCV 4 Android.
Now you have a tool to detect objects in Android AR apps.
On 8th of May 2018 Google announced a mobile SDK – ML Kit, that's a machine learning API for Android and iOS developers that you can use along with ARCore SDK (what now has Augmented Images API and Augmented Faces API). At the moment it's still in beta stage but it's got the following working features:
ML Kit engineers say about it this:
ML Kit brings Google's machine learning expertise to Android and iOS apps in a powerful yet easy-to-use package. Whether you're new or experienced in machine learning, you can implement the functionality you need in just a few lines of code. There's no need to have deep knowledge of neural networks or model optimization to get started. On the other hand, if you are an experienced ML developer, ML Kit provides convenient APIs that help you use your custom TensorFlow Lite models in your mobile apps.

Resources