Saturday, 29 June 2019

Converting ImageProxy to Bitmap

So, I wanted to explore new Google's Camera API - CameraX. What I want to do, is take an image from camera feed every second and then pass it into a function that accepts bitmap for machine learning purposes.

I read the documentation on Camera X Image Analyzer:

The image analysis use case provides your app with a CPU-accessible image to perform image processing, computer vision, or machine learning inference on. The application implements an Analyzer method that is run on each frame.

..which basically is what I need. So, I implemented this image analyzer like this:

imageAnalysis.setAnalyzer { image: ImageProxy, _: Int ->
    viewModel.onAnalyzeImage(image)
}

What I get is image: ImageProxy. How can I transfer this ImageProxy to Bitmap?

I tried to solve it like this:

fun decodeBitmap(image: ImageProxy): Bitmap? {
    val buffer = image.planes[0].buffer
    val bytes = ByteArray(buffer.capacity()).also { buffer.get(it) }
    return BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
}

But it returns null - because decodeByteArray does not receive valid (?) bitmap bytes. Any ideas?



from Converting ImageProxy to Bitmap

Node.js - FCM send scheduled topic notification

I'm using an Angular app and a Node.js server to send topic notifications to Android devices using FCM. I am using bull to schedule the notifications.

There are two ways of sending a notification: send a notification now and send a notification at a specific date and time.

To send notifications I am using an Angular app and this method:

  sendTopicNotification(notification: Notification): Observable<any> {
    console.log("send topic");
    return this.http.post(endpoint + `notifications/send/topic/topic${notification.target}`, JSON.stringify(notification), httpOptions).pipe(
      map(result => { return true; })
    );
  }

This is my Node.js code:

const express = require('express');
const router = express.Router();
const fb = require('../firebase/fb');
const db = fb.firestore();
const fcm = fb.messaging();
const moment = require('moment');

var Queue = require('bull');
var notificationsQueue = new Queue('topic notifications', { redis: { port: 6379, host: '127.0.0.1' } }); // Specify Redis connection using object

const notificationsCollection = 'notifications';
const donationsCollection = 'donations';


router.post('/send/topic/:topic', (req, res) => {
    var topic = `/topics/${req.params.topic.toString()}`;

    var payload = {
        notification: {
            title: req.body.title,
            body: req.body.body
        }
    };

    var options = {
        priority: "high",
        timeToLive: 60 * 60 * 24
    };

    if (req.body.sendDate && req.body.sendHour) {
        var date = req.body.sendDate;
        var hour = req.body.sendHour;
        scheduleMessage(date, hour, topic, payload, options);
    } else {
        sendTopicNotification(topic, payload, options);
    }


    res.send(200);
});

//Schedule the job
async function scheduleMessage(date, hour, topic, payload, options, res) {
    var date = date.toString().split("/");
    var hour = hour.toString().split(":");
    console.log(date[2], date[1], date[0], hour[0], hour[1], 0);
    var jobDate = new Date(date[2], date[1] - 1, date[0], hour[0], hour[1]);

    console.log(jobDate);
    console.log(new Date());
    var jobDelay = ((jobDate.getTime() / 1000) - (Math.floor(new Date().getTime() / 1000)));

    console.log(jobDate.getTime() / 1000);
    console.log(Math.abs(jobDelay));
    console.log(Math.floor(new Date().getTime() / 1000));

    const job = await notificationsQueue.add({
        topic: topic,
        payload: payload,
        options: options
    }, { delay: Math.abs(jobDelay) });
    console.log(date + " " + hour);
}

//Process qued job
notificationsQueue.process(async (job, done) => {
    console.log(job.data);
    sendTopicNotification(job.data.topic, job.data.payload, job.data.options);
});

//Send notificaiton
function sendTopicNotification(topic, payload, options) {
    var currentTime = new Date().getTime();

    var target;
    switch (topic) {
        case "/topics/topicA":
            target = 'Donatorii cu grupa sanguină A'
            break;
        case "/topics/topicB":
            target = 'Donatorii cu grupa sanguină B'
            break;
        case "/topics/topicAB":
            target = 'Donatorii cu grupa sanguină AB'
            break;
        case "/topics/topic0":
            target = 'Donatorii cu grupa sanguină 0'
            break;
        case "/topics/topicAll":
            target = 'Toți donatorii'
            break;
        default:
            break;
    }
    fcm.sendToTopic(topic, payload, options)
        .then((response) => {
            db.collection(notificationsCollection).doc(currentTime.toString()).set({
                title: payload.notification.title,
                body: payload.notification.body,
                date: currentTime,
                target: target,
                status: "Notificarea a fost trimisă!"
            }).then((res) => {
                console.log('Create new notification ');
            });
            // Response is a message ID string.
            console.log('Successfully sent message:', response);
        })
        .catch((error) => {
            db.collection(notificationsCollection).doc(currentTime.toString()).set({
                title: payload.notification.title,
                body: payload.notification.body,
                date: currentTime,
                target: topic,
                status: "Notificarea nu a fost trimisă!"
            }).then(() => {
                console.log('Create new notification');
            });
            console.log('Error sending message:', error);
        });
}
module.exports = router;

1). "Now" notification

Notification body:

{
    body: "test",
    date: undefined,
    status: "Notificarea nu a fost trimisă!",
    target: "A",
    title: "test",
}

These are the result and the logs:

express deprecated res.send(status): Use res.sendStatus(status) instead routes/notifications.js:41:9
Successfully sent message: { messageId: 9203836031271870000 }
Create new notification

The notification reaches the Android app and everything works normal as expected.

2). "Scheduled notification"

Notification body:

{
    body: "test",
    date: undefined,
    sendDate: "29/06/2019",
    sendHour: "07:27",
    status: "Notificarea nu a fost programată!",
    target: "A",
    title: "test",
}

These are the result and the logs:

2019-06-29T04:27:00.000Z
2019-06-29T04:25:35.070Z
{ topic: '/topics/topicA',
  payload: { notification: { title: 'test', body: 'test' } },
  options: { priority: 'high', timeToLive: 86400 } }
Successfully sent message: { messageId: 5284791767401410000 }
Create new notification 

Now there is something wrong with the behaviour. The notification won't reach the Android app until I restart the server. As you can see it is programmed to be sent at 2019-06-29T04:27:00.000Z, but nothing was sent even after a longer period. When I restart the server the Android app will receive the last sent notification.

So my problems are in the second scenario:

  • notifications are not sent after the delay;
  • the notifications seem not to be stored in a queue so only the last one will be sent after I restart the server;
  • after I restart the server the previously scheduled notifications seem to be sent on and on. Eg: first notification is sent, restart, second is sent, restart, first is sent, etc.

What am I missing?



from Node.js - FCM send scheduled topic notification

iOS app -- no cellular access to our domain on some devices

With a particular React Native app, some iPhone users are experiencing an issue where the app can almost never make web requests to our API when connected via cellular data. The domain that is having issues points to an Amazon Elastic Load Balancer, which points to an Nginx reverse proxy. Other APIs (e.g. Mapbox) called by the app work fine over cellular data, including one of ours hosted on a dedicated server, just not those on our ELB domain. When the user switches to WiFi, our app is able to make web requests to that domain. This has been observed on iPhone 7, iPhone 8, and iPhone X, all running iOS 12.3.1. One device is Verizon and the other 4 reported are AT&T. Every API call is HTTPS. Deleting and reinstalling the app and restarting the device does not resolve the issue. We confirmed in all cases that cellular data was enabled for the app in Settings > Cellular > [App name] and in Settings > [App name] > Use Cellular Data.

The app is built with React Native and web requests are performed with the cross-fetch library.

We were able to get a device that has the issue and run it through Xcode. Here is a subset of the error stack captured in Xcode:

nw_connection_copy_connected_local_endpoint [C12] Connection has no local endpoint
2019-06-27 11:26:16.841347-0400 myapp[23700:1527268] [BoringSSL] 
nw_protocol_boringssl_get_output_frames(1301) [C10.1:2][0x117d5a050] get output frames failed, state 8196

2019-06-27 11:26:22.465855-0400 myapp[23700:1527305] [BoringSSL] nw_protocol_boringssl_error(1584) [C20.1:2][0x119b0e420] Lower protocol stack error: 54
2019-06-27 11:26:22.466665-0400 myapp[23700:1527305] TIC TCP Conn Failed [20:0x280022400]: 1:54 Err(54)

2019-06-27 11:26:23.040101-0400 myapp[23700:1527399] Task <DD5FDD4A-1BE0-41ED-AAC4-9EB07F61F109>.<7> HTTP load failed (error code: -1005 [1:54])
2019-06-27 11:26:23.040408-0400 myapp[23700:1527305] Task <DD5FDD4A-1BE0-41ED-AAC4-9EB07F61F109>.<7> finished with error - code: -1005
load failed with error Error Domain=NSURLErrorDomain Code=-1005 "The network connection was lost." UserInfo={_kCFStreamErrorCodeKey=54, NSUnderlyingError=0x283a521f0 {Error Domain=kCFErrorDomainCFNetwork Code=-1005 "(null)" UserInfo={NSErrorPeerAddressKey=<CFData 0x28161ab70 [0x1e9e5d420]>{length = 16, capacity = 16, bytes = 0x100201bb3416ca8a0000000000000000}, _kCFStreamErrorCodeKey=54, _kCFStreamErrorDomainKey=1}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <DD5FDD4A-1BE0-41ED-AAC4-9EB07F61F109>.<7>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalDataTask <DD5FDD4A-1BE0-41ED-AAC4-9EB07F61F109>.<7>"
), NSLocalizedDescription=The network connection was lost.

Queries to this particular ELB -> Nginx -> Kubernetes services setup will occasionally work but then stop. It almost indicates a keep-alive situation like this issue. We had the ELB idle timeout set at its default (60s) and we increased it to 300s with no apparent effect. We tried with the keep-alive for Nginx both set to 360s and with it disabled completely.

For the domain in question we have a mix of services hosted in the Kubernetes cluster, such as Java and Node.js. The issue affects all of them equally.

None of the Android app users have reported this issue.

The devices that experience this issue all do so consistently, it is not intermittent.

Due to the type of error, the requests never reach our Nginx logs.



from iOS app -- no cellular access to our domain on some devices