This sample projects shows how to "crop" the environment using hand gesture.
NOTE: This project will only work for the Spectacles platform.
Designing Lenses for Spectacles offers all-new possibilities to rethink user interaction with digital spaces and the physical world. Get started using our Design Guidelines
- Lens Studio: v5.15.0+
- Spectacles OS Version: v5.64+
- Spectacles App iOS: v0.64+
- Spectacles App Android: v0.64+
To update your Spectacles device and mobile app, please refer to this guide.
You can download the latest version of Lens Studio from here.
The camera feature requires you to use Experimental APIs. Please see Experimental APIs for more details here.
Extended Permissions mode on device must be enabled for enabling some of the Spectacles APIs. Please see Extended Permissions for more details here.
To obtain the project folder, you need to clone the repository.
IMPORTANT: This project uses Git Large Files Support (LFS). Downloading a zip file using the green button on Github will not work. You must clone the project with a version of git that has LFS. You can download Git LFS here: https://git-lfs.github.com/.
In order to use this project and call Remote Service Gateway APIs, you need to:
- Install the Remote Service Gateway Token Generator plug-in from the Asset Browser
- Go to Window -> Remote Service Gateway Token
- Click "Generate Token"
- Copy paste the token into the "RemoteServiceGatewayCredentials" object in the Inspector
The camera module allows access to the left or right camera texture on device. To align the image crop with either camera we can use the camera module to get intrinsics and pose (offset) of each camera. We can then create two virtual cameras underneath the main camera and set their respective physical properties. This way we can get camera space positions of objects in 3D space. This project uses those positions to update the screen crop texture accordingly.
There is CropRegion.ts that takes 4 SceneObjects and uses those for the corners of the crop. That could be used to crop a fixed area anchored to the hand or some fixed distance from the camera.
Use the interactive preview to move around and click the screen to test the cropping behavior and ChatGPT functionality.
Install the lens as normal and pinch both hands close together, pull your right hand diagonally down to size the capture window, release the pinch on both hands to send capture to ChatGPT.
Ensure that you comply with OpenAI’s API usage policies and Spectacles’ terms of service when deploying this project.
If you have any questions or need assistance, please don't hesitate to reach out. Our community is here to help, and you can connect with us and ask for support here. We look forward to hearing from you and are excited to assist you on your journey!
Feel free to provide improvements or suggestions or directly contributing via merge request. By sharing insights, you help everyone else build better Lenses.
Built with 👻 by the Spectacles team