Building Software That Scales

53646549

It’s been a long time since I’ve posted here.  I’ve been really busy in general, but I’m excited to publish this post (I actually forced myself to publish this post).

Over the past few weeks, since I returned from vacation, I’ve been working on building a backend that scales.  I’m building an Android podcast app, PrēmoFM.  My podcast app needs a backend parsing infrastructure that scales to parse hundreds of thousands of podcast XML files constantly.  As I type this, I’m about to unleash hundreds of thousands of podcast channels onto my beta users.  Up to this point, they’ve only had access to 75, which mostly consisted of my favorite podcasts.  That number will be increasing to north of 200,000.  I’m ecstatic about this because these are the first pieces of software that I built that operates at this scale.

In order to provide a good user experience to PrēmoFM users, I need to be able to parse at least 5,000 podcasts per minute.  On the surface, this would allow me to run through the entire catalog one time in 40 minutes.  I’ve made several architectural decisions that allow me to achieve this.  I’ve built highly multi-threaded Java apps that serve a single purpose:

  • HTTP XML data retrievers
  • XML Parsers
  • Push message senders

Each portion of the chain has it owns challenges and constraints, so I needed to build apps that were focused in their function.  This also allows me to spin up instances of each app to keep my XML parsing pipeline saturated and operating at peak performance, which is exactly what I did.  Secondarily, I’m able to iterate and update each component separately.  I can only imagine the frustration if I had one Java app that did the retrieval, XML parsing, and push message sending.  I’m still doing to a bit of testing before I move everything up to my production VPSes (virtual private servers), but I’ve already blew past my baseline goal of 5,000 podcasts per minute and am peaking around 12,000+ (Update 5/25/2015 @ 1AM – deployed to DigitalOcean – peaking at 27,000 channels per min, bonkers, to me anyway), which is at least two orders of magnitude increase from where I was a week ago.  In fact, things are moving so fast, I’m going to have to significantly cut down on the logging or it’ll begin eating up all of the available hard drive space.

I’ll provide more detail on my backend architecture in the future, when I have time to document.  For right now, I’m marching towards release.

Are you an Android user?  Sign up for the beta at Prēmo.FM

Podcasts by the Numbers

I’m building a mobile podcast “catcher” app (called Prēmo – GetPremo.com, pronounced Preeemo).  Midroll, a company that connects podcasters with advertisers, publishes some interesting results from their quarterly survey, which has 279K responses from 106 podcast shows.

Just a few interesting statistics:

  • 58% of podcast listeners have at least a Bachelor’s degree
  • 80% of podcast listeners listen using their mobile device (iPhone, Android, other)
  • 67% of podcast listeners are in the 18-34 year old demographic (ie. the most desirable demographic)

That was just a small sample of some amazing and insightful statistics published by Midroll.

Check the rest out for yourself.

Muneee Mondays Vol. 2 Episode 2

New Muneee Mondays podcast available.  DOWNLOAD Tracklist:

  1. Drake “Over”
  2. T.I. “I’m Back”
  3. Vocal Kid “I’m Good”
  4. Actual Proof “Genius”
  5. GQ “Lights, Camera, Action”
  6. David Banner ft. Big Remo “Stranger”
  7. Young Chris ft. J. Cole “Still The Hottest”
  8. Sheek Louch ft. Red Cafe “Ain’t No Half Steppin’ 2010”
  9. ArtOfficial ft. K. Sparks “The Payback” <——my favorite track this podcast, google these guys, dope
  10. Ayah “He Don’t Want It”
  11. Musiq Soulchild “Forthenight”
  12. Common “I Want You”
  13. Goodie Mobb ft. Outkast “Sky High”
  14. Will Smith Pickup Lines

Shout’s to DJ Statik on the drops.  When will the next episode drop? Monday? lol, we’ll see.

http://138.197.109.115/blog/DJ/music/Muneee_Mondays_Vol_2_Ep_2.mp3

Muneee Mondays Vol. 2 Episode 1

It’s been at least four months since I dropped a Muneee Mondays episode.  The last four months of 2009 was ridiculously busy to say the least.  However, I come bearing gifts on the first Monday of 2010 for everyone going back to work after the holiday season.

Tracklist:

  1. Talib Kweli ft. Termanology & Royce Da 5’6 “Come Around”
  2. Nio That Gift “Grateful” (I’m feeling this record)
  3. Kanye West “I’m So Appalled”
  4. Reflection Eternal ft. Jay Electronica, J. Cole, Mos Def “Just Begun” (no hook, just rhyme spitting, real dope)
  5. Common “Dooinit”
  6. King Mez of The Lazaretto Crew “Lookwarm” (http://welcometomezatron.blogspot.com)
  7. Big Pun ft. The Beatnuts “Off The Books”
  8. B.o.B. aka Bobby Ray ft. Bruno Mars “Nothing On You”‘

You can stream it using the Stream Pad at the bottom of the window (ahh Technology).

Download -> http://138.197.109.115/blog/DJ/music/Muneee_Mondays_Vol_2_Ep_1.mp3

King Mez & Commisioner Gordon Collab Album “Paraplegics” drops in Q1 2010.  Look out for that.