Makes an HTTP request.

Factory function for AsyncStorage IOHandler.

This IOHandler supports both save and load.

For each model's saved artifacts, three items are saved to async storage.

• tensorflowjs_models/${modelPath}/info: Contains meta-info about the model, such as date saved, type of the topology, size in bytes, etc.
• tensorflowjs_models/${modelPath}/model_without_weight: The topology, weights_specs and all other information about the model except for the weights.
• tensorflowjs_models/${modelPath}/weight_data: Concatenated binary weight values, stored as a base64-encoded string.

  async function asyncStorageExample() {
    // Define a model
    const model = tf.sequential();
    model.add(tf.layers.dense({units: 5, inputShape: [1]}));
    model.add(tf.layers.dense({units: 1}));
    model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});

    // Save the model to async storage
    await model.save(asyncStorageIO('custom-model-test'));
    // Load the model from async storage
    await tf.loadLayersModel(asyncStorageIO('custom-model-test'));
}

Factory function for BundleResource IOHandler.

This IOHandler only supports load. It is designed to support loading models that have been statically bundled (at compile time) with an app.

This IOHandler is not compatible with managed expo apps.

  const modelJson = require('../path/to/model.json');
  const modelWeights = require('../path/to/model_weights.bin');
  async function bundleResourceIOExample() {
    const model =
      await tf.loadLayersModel(bundleResourceIO(modelJson, modelWeights));

     const res = model.predict(tf.randomNormal([1, 28, 28, 1])) as tf.Tensor;
  }

Decode a JPEG-encoded image to a 3D Tensor of dtype int32.

// Load an image as a Uint8Array
const imageUri = 'http://image-uri-here.example.com/image.jpg'; *
const response = await fetch(imageUri, {}, { isBinary: true });
const imageDataArrayBuffer = await response.arrayBuffer();
const imageData = new Uint8Array(imageDataArrayBuffer);

// Decode image data to a tensor
const imageTensor = decodeJpeg(imageData);

A higher-order-component (HOC) that augments the Expo.Camera component with the ability to yield tensors representing the camera stream.

Because the camera data will be consumed in the process, the original camera component will not render any content. This component provides options that can be used to render the camera preview.

Notably the component allows on-the-fly resizing of the camera image to smaller dimensions, this speeds up data transfer between the native and javascript threads immensely.

In addition to all the props taken by Expo.Camera. The returned component takes the following props

• use_custom_shaders_to_resize: boolean — whether to use custom shaders to resize the camera image to smaller dimensions that fit the output tensor.
  • If it is set to false (default and recommended), the resize will be done by the underlying GL system when drawing the camera image texture to the target output texture with TEXTURE_MIN_FILTER/TEXTURE_MAG_FILTER set to gl.LINEAR, and there is no need to provide cameraTextureWidth and cameraTextureHeight props below.
  • If it is set to true (legacy), the resize will be done by the custom shaders defined in resize_bilinear_program_info.ts. Setting it to true also requires that client provide the correct cameraTextureWidth and cameraTextureHeight props below. Unfortunately there is no official API to get the camera texture size programmatically so they have to be decided empirically. From our experience, it is hard to cover all cases in this way because different devices models and/or preview sizes might produce different camera texture sizes.
• cameraTextureWidth: number — the width the camera preview texture (see note above)
• cameraTextureHeight: number — the height the camera preview texture (see note above)
• resizeWidth: number — the width of the output tensor
• resizeHeight: number — the height of the output tensor
• resizeDepth: number — the depth (num of channels) of the output tensor. Should be 3 or 4.
• autorender: boolean — if true the view will be automatically updated with the contents of the camera. Set this to false if you want more direct control on when rendering happens.
• rotation: number — the degrees that the internal camera texture and preview will be rotated. Accepted values: 0, +/- 90, +/- 180, +/- 270 or 360.
• onReady: ( images: IterableIterator<tf.Tensor3D>, updateCameraPreview: () => void, gl: ExpoWebGLRenderingContext, cameraTexture: WebGLTexture ) => void — When the component is mounted and ready this callback will be called and recieve the following 3 elements:
  • images is a (iterator)[https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators] that yields tensors representing the camera image on demand.
  • updateCameraPreview is a function that will update the WebGL render buffer with the contents of the camera. Not needed when autorender is true
  • gl is the ExpoWebGl context used to do the rendering. After calling updateCameraPreview and any other operations you want to synchronize to the camera rendering you must call gl.endFrameExp() to display it on the screen. This is also provided in case you want to do other rendering using WebGL. Not needed when autorender is true.
  • cameraTexture The underlying cameraTexture. This can be used to implement your own updateCameraPreview.

import { Camera } from 'expo-camera';
import { cameraWithTensors } from '@tensorflow/tfjs-react-native';

const TensorCamera = cameraWithTensors(Camera);

class MyComponent {

   handleCameraStream(images, updatePreview, gl) {
     const loop = async () => {
       const nextImageTensor = images.next().value

       //
       // do something with tensor here
       //

       // if autorender is false you need the following two lines.
       // updatePreview();
       // gl.endFrameEXP();

       requestAnimationFrame(loop);
     }
     loop();
   }

   render() {
    return <View>
      <TensorCamera
       // Standard Camera props
       style={styles.camera}
       type={Camera.Constants.Type.front}
       // Tensor related props
       resizeHeight={200}
       resizeWidth={152}
       resizeDepth={3}
       onReady={this.handleCameraStream}
       autorender={true}
      />
    </View>
   }
}

Utility function that tests the GL context for capabilities to enable optimizations.

For best performance this should be be called once before using the other camera related functions.

Creates a tensor3D from a texture.

Allows for resizing the image and dropping the alpha channel from the resulting tensor.

Note that if you the output depth is 3 then the output width should be a multiple of 4.

Render a texture to the GLView. This will use the default framebuffer and present the contents of the texture on the screen.

Transfers tensor data to an RGB(A) texture.