A Journey Towards Android App Log Subscriptions

I’ve got an Android project going that uses FFmpeg to build short video files (more on this at a later date). Normally, you run FFmpeg in a terminal window or command prompt.  It updates you on the status of a task in that terminal window.  I’ve instrumented FFmpeg on Android with some C code and a JNI interface.  When I execute FFmpeg from an Android app those status updates are written to logcat.

There are some important details that I was to surface in my app, primarily the progress of an encoding process.  In order to do this, I needed a way to read my Android apps logging statements.  This is possible on Android and without needing to request the scary READ_LOGS Android permission.

Note: All of my examples are written in Kotlin because I ❤ Kotlin.

You can execute a process with the Runtime API:

val process = Runtime.getRuntime().exec("some_command_here")

You can then get an InputStream to the Process’s output:

val process = Runtime.getRuntime().exec("some_command_here")
val inputStream = process.inputStream

Putting it all together, this is how one would get your apps logging statements from logcat, within the scope of your app:

val process = Runtime.getRuntime().exec("logcat")
reader = BufferedReader(InputStreamReader(process.inputStream))
var line: String?

do {
    line = reader.readLine()

    if (line != null) {
        // do something with this logging statement
    }
} while (line != null)

reader.close()

This code essentially runs forever because, as far as I’ve seen, reader doesn’t return a null line.

I’m close, but I still needed to add some sophistication. I needed to read and parse logs while FFmpeg was running so I can update the user on the status of their job. I needed to start parsing logs when I wanted to update the user while FFmpeg was running and end parsing when FFmpeg was complete. This problem is perfectly solved using RxJava:

fun readLogs(): Observable = 
  return Observable.create ({ emitter ->
    var reader: BufferedReader? = null

    try {
      val process = Runtime.getRuntime().exec("logcat")
      reader = BufferedReader(InputStreamReader(process.inputStream))
      var line: String?

      do {
          line = reader.readLine()

          if (line.isNotBlank()) {
              emitter.onNext(line)
          }
      } while (line != null)
      emitter.onComplete()

    } catch (e: Exception) {
      emitter.onError(e)
    } finally {
      reader?.close()
    }
  })

With RxJava, I can emit a logging statement as it’s received by logcat.

Usage:

val disposable = readLogs()
  .subscribeOn(Schedulers.io())
  .observeOn(AndroidSchedulers.mainThread())
  .subscribe({ logLine -> 
    logTextView.setText(logLine)
  }, { error -> throw error })
  
// later, when you no longer need to subscribe to your logs
disposable.dispose()

Because it’s all based on RxJava, you can do more advanced things like filtering

val disposable = readLogs()
  .filter({ line -> line.startsWith("foo") })
  .subscribeOn(Schedulers.io())
  .observeOn(AndroidSchedulers.mainThread())
  .subscribe({ logLine -> 
    logTextView.setText(logLine)
  }, { error -> throw error })

This allows you to “subscribe” to your apps logs!

/* fini */

// The Comment #12: Crawling, Downloading, Parsing, Notifying

Durm

The Comment is a weekly digest of the stuff that grabbed my attention or occupied some part my mind during the past week. Normally, it’ll be one thing that’s really been on my mind, followed by a handful of things that I found interesting. The Comment will be published each Monday at 10:30AM EST. 

Thanks for reading.

# Open sourced some infrastructure

In 2014 to 2015, I spent a lot of my spare time building a podcast app for Android. I probably spent hundreds of hours coding the Android app and the parsing / syncing backend.  I shut the app down a few months after I released it because I failed to quickly find a good “product market fit”.  Anyway, I open sourced the Android app a year ago or so and it even runs as a standalone Android app (no backend needed).

Today I’m open sourcing the backend apps and APIs that ran everything.  There’s nothing spectacular here.  It’s just a bunch of Java and Node.JS code.  There are several apps here:

  • PodcastCrawler – Crawled iTunes podcasts for all the podcasts.
  • Retriever – Pulled podcasts RSS urls from the database, retrieved the feeds data, and stuck the data into a Redis queue.
  • Parser – Parsed feed data from a Redis Queue and stuck Podcast and Episode objects into another Redis queue.
  • Storer – Wrote Podcast and Episode records into the database.  It also built notification records (for push messaging) and stuck those into a Redis queue.
  • Notifier – Sent Google Cloud Messages to devices for push-to-sync purposes.
  • Billboard – Built the trending podcast lists by crunching user listen data.
  • API – Source of all podcast and episode data as well as an avenue for account creation and syncing

Looking back, this was the first large scale backend I built by myself.  It parsed 200k+ podcasts, multiple times an hour, making available millions of podcast episodes.  I learned all about DevOps, setting up VPN, scripting deployments, and the MEAN stack while working on this project.

Check the project out on GitHub.

// The Comment #11: No Distract Zone

The Comment is a weekly digest of the stuff that grabbed my attention or occupied some part my mind during the past week. Normally, it’ll be one thing that’s really been on my mind, followed by a handful of things that I found interesting. The Comment will be published each Monday at 10:30AM EST. 

Thanks for reading.

# Eliminating Distraction

Most of us have smartphones loaded with tons of apps used for communicating, news, social media, and entertainment. This presents tons of distractions. Fortunately, we have total control over how we let smartphone apps interfere our real lives. This article at timewellspent.io gives us a few tips for reducing or eliminating this distraction.

Some things I’ve done to eliminate some distractions:

  1. Remove social media apps. I don’t use apps for Twitter or Facebook. I sign into Twitter’s mobile website. I try not to sign into Facebook at all. The only social media app I use is Instagram.
  2. Turn off useless notifications. I only get interrupted by phone calls and direct messages (text & chat). I have turned off Instagram likes. I get notifications for breaking news, weather, calendar invites, NY Jets losses, and reminders.
  3. One-two tap access is reserved for apps that allow me to do constructive things like writing or passive entertainment like listening to music. I have to go to my app drawer for everything else.

// Injustice in the US criminal justice system

I listened the following TED Talk thinking it was published recently. It was published in 2012. This is an evergreen TED talk.

Bryan Stevenson talks about the injustice in the US criminal justice system.

// “Beyond the Bitcoin Bubble”

An incredible article by Steven Johnson at the New York Times.

For our purposes, forget everything else about the Bitcoin frenzy, and just keep these two things in mind: What Nakamoto ushered into the world was a way of agreeing on the contents of a database without anyone being “in charge” of the database, and a way of compensating people for helping make that database more valuable, without those people being on an official payroll or owning shares in a corporate entity. Together, those two ideas solved the distributed-database problem and the funding problem. Suddenly there was a way of supporting open protocols that wasn’t available during the infancy of Facebook and Twitter.

Blockchain is one of the most revolutionary technologies invented in some time.

/* fini */

Chromecast, the Pixel 2, and MDNS

There have been widespread reports of some network issues for users with a certain brand of consumer router, Chromecast / Chromecast enabled devices, and Pixels.  I’ve been having sporadic WiFi issues for a few weeks to a month.  I finally but 2 and 2 together. I have a number of Google devices in my house (a lot..perhaps too many 🙂 ).

  • Google WiFi mesh router system
  • 2 Chromecast audios
  • 3 Chromecast dongles (2 1st gen, 1 2nd gen)
  • 2 Android TV devices, Nexus Player running Android Oreo and a Xiaomi MiBox running Android 6
  • 3 Google Homes, 2 normal, 1 mini

I also use a Google Pixel 2 XL as my main device. I have several test devices as well. A problem I’ve been having for a while now is when I wake my Pixel up I lose my WiFi connection. It happens very sporadically. It’s annoying. There are also times when my Nexus Player loses its WiFi connection. In both instances, the WiFi connection is re-established, but it makes things like streaming video or controlling streaming video (via Chromecast) a nuisance.

I’m also an Android developer by day and have ADB Logcat open at all times. I use a Nexus 6 running Android 7.1.1 as my primary test device. Periodically, I’ll see the following group of log statements printed in Logcat, for my Nexus 6.

I/DeviceScanner: [MDNS] Received response from "Cast Device 1" (-----------) with result "0 No change"
I/DeviceScanner: [MDNS] Received response from "Cast Device 2" (-----------) with result "0 No change"
I/DeviceScanner: [MDNS] Received response from "Cast Device 3" (-----------) with result "0 No change"
I/DeviceScanner: [MDNS] Received response from "Cast Device 4" (-----------) with result "0 No change"
I/DeviceScanner: [MDNS] Received response from "Cast Device 5" (-----------) with result "0 No change"
I/DeviceScanner: [MDNS] Received response from "Cast Device 6" (-----------) with result "0 No change"

Note: I did mask my device names and MAC addresses in this snippet.

These are MDNS requests being logged by my Nexus 6 as they are received. I don’t have significant network experience or a deep understanding on how Chromecast really works at the network level. It appears, to me, that every 20 seconds or so, all of my Chromecast and Chromecast enabled devices send an MDNS packet announcing their presence on my network to devices that can utilize them, like Android phones and Google Home.

In fact, TP-Link, a router manufacturer, has come to the following conclusion:

This issue stems from these devices’ “Cast” feature, which sends MDNS multicast discovery packets in order to discover and keep a live connection with Google products such as Google Home. These packets normally sent in a 20-second interval. However, we have discovered that the devices will sometimes broadcast a large amount of these packets at a very high speed in a short amount of time. This occurs when the device is awakened from its “sleep” state, and could exceed more than 100,000 packets. The longer your device is in “sleep”, the larger this packet burst will be. This issue may eventually cause some of router’s primary features to shut down – including wireless connectivity.

So obviously there’s a problem here. Today I noticed, when I power on my Pixel 2 XL running Android 8.1 (w/ January security update), things get pretty crazy. My Nexus 6 seems to log over 1,000 MDNS responses in a 40 second timespan. With my limited networking knowledge, I don’t really know if anything is wrong, but it seems that something is out of wack. It also convenient that I tend to drop my WiFi connection, on my Pixel 2 XL at this time (when my Pixel is powered on from sleeping).

Running Wireshark on a computer verifies what I see in Logcat.  A gargantuan amount of MDNS activity upon powering on my Pixel can be seen.  It also stops when I put my Pixel back to sleep.  Waking my Nexus 6 up doesn’t result in the same amount of MDNS activity.

Something is probably wrong here.  I can’t identify it specifically, but this is just another data point.

// The Comment #10: I should be doing more

The Comment is a weekly digest of the stuff that grabbed my attention or occupied some part my mind during the past week. Normally, it’ll be one thing that’s really been on my mind, followed by a handful of things that I found interesting. The Comment will be published each Monday at 10:30AM EST. 

Thanks for reading.

## I should be doing more

This a short entry this week and a little personal. It’s a weird feeling to know you can do more, but you’re not, and you know that. I should be doing more and having more of a positive impact on society and culture. On the basis of using my talents and abilities to lift up the next person who may not have the same.  I’m talking about my abilities to write code, put things on the Internet, general handy-ness, time, attention, and effort.

I came to his conclusion recently, but I have been heading down this mental road for a while.  This is something I’m going to resolve in 2018.

// More North Carolina Districting Shenanigans

From The Washington Post:

A federal court on Tuesday ruled that Republicans in North Carolina unconstitutionally gerrymandered congressional districts in 2016 to ensure Republican “domination of the state’s congressional delegation.”

The three-judge panel struck down the map and ordered the state’s General Assembly to come up with a substitute by Jan. 24.

The decision was the first striking down of a congressional map, as opposed to a state legislative map, on the grounds that it was rigged in favor of a particular political party. Redistricting has historically been political and partisan to one degree or another.

Oh look, a federal court struck down another set of maps drawn up by the North Carolina Republican Party. This time, it was the congressional maps…the ones that determine the districts of our Congressional representatives.  As a resident of North Carolina, its getting a bit ridiculous that Republicans continue to draw gerrymandered legislative (and soon to be judicial) maps.  It seems like the kind of thing you do when you know your time in power is coming to an end.

// A Spectre and a Meltdown Walk Into a Bank

Paul Miller writing for The Verge:

You walk up to the door and you don’t go into the bank. In the parallel reality where you do go into the bank, you enter the vault and look at the piece of paper. You read the password and whisper it quietly before you get shot dead.

In the reality where you don’t go into the bank, you own a highly elaborate listening device which can hear your parallel self’s whispers. Now you know Ashley’s Netflix password, and can enjoy all manner of original content at her expense.

This banking analogy is one of the more accessible explanations on what the Spectre and Meltdown bugs are. Spectre and Meltdown are a set of bugs that affect a ton of computers and other devices running Intel, AMD, and ARM processors. As someone who’s actually studied processor design in my college days, many many years ago, I cannot explain why Spectre and Meltdown are such a big deal, but they are.

/* fini */