We woke up yesterday with smiles on our faces and using our favorite chrome plugin (Parrotfish) . Then we got the news that the #new#new Twitter was released. Jaws dropped, tweets flew in, and profanities flew out. Our users expected results. @hotdogsladies tweeted that lovemaking was just not the same without Embedly. We concur.

For a brief moment we considered retiring Parrotfish. Surely in this latest release Twitter would have implemented embeds the way they should have from the beginning. Lucky for us, it appears to be the same crippled system that caused us to create Parrotfish in the first place.

So, off to the Batcave Sean and Bob went. Afterall, we do our best work with bats circling. Who doesn't? They (Sean and Bob, not the bats) woke up this morning, ready again to tread through the depths of a Twitter re-design, this time armed with some new toys that we have created over the last few months.

We now present to you the latest and greatest Parrotfish ready to conquer your timeline (the Twitter one, not the Facebook one):

•  Enabled with SSL support for embeds and images. (Secure)
•  Better favicons and logos.
• Available in Chrome and Safari. (FF you're next)

Get it right away at Embedly Labs.

Last week we took a break from bug fixing, redesigning, and development. We held our own internal Hack Week: 4 developers, 4 completely different projects, all using or enhancing the Embedly service.

Arthur spent the week adding more social features to Embedly. We want to be able to answer the question: "what's the most popular content on my site?" Arthur developed a Reddit-like voting system for embeds, that get tallied by us and displayed with the rest of our Analytics.

Bob created a web socket proxy for the Embedly API, developed using node.js, because Bob loves node.js. The proxy allows for truly asynchronous requests to the Embedly API, returning embeds as they finish instead of all at once. If anyone is interested, Bob will add documentation when he has some free time, probably during the next Embedly Hack Week.

Sean, the master of Chrome plugins, developed a super top secret Chrome plugin. We could tell you about it, but then we'd have to kill you. Or we could make you sign an NDA, but paperwork is messy.

We sometimes get too focused on business development, answering support tickets, and making sure the servers stay up and running. It's nice every once in a while to take a step back and reap what we've sown. We're constantly surprised with what we've managed to accomplish over the last two years.

We love to hear how others are using the Embedly API. Let us know in comments, and as always, we're available at support@embed.ly with any questions.

In our opinion censorship on the web is never a good thing. While we are not here to get on the soapbox or try to instill values, SOPA is just bad for business. 

Embedly is a company that deals with millions of links a day. If you start censoring what people can link to, inevitably that hurts us. 

When a user embeds an infringing video, who is at risk? The site that embedded the video, the video hosting service or the delivery mechanism? I certainly don't want to find out. Maybe it will be like prohibition and Embedly will become the bootleggers of our time, transporting content from provider to publisher.

If you visit embed.ly today you will see a tiny band over our logo in solidarity for the cause.

We encourage you to read the bill, watch the video and visit americancensorship.org to learn more.

Sean

In this post we will take an in depth look at how Embedly tracks API usage. Warning: this post is not for the faint of heart.

Overview

Our API consists of a cluster of tornado boxes behind two nginx load balancers. API usage is based on the nginx access log at the load balancer. This allows us to seperate tracking from the API.

Nginx logs are transported from the load balancer to the log queue. The log queue is simply a Redis list. Logs are transported to the queue with logstash. Logstash also parses the access logs into structured data, making it easier to process them later. We could have written all of our log processing as a logstash filter, but we have a lot of existing code that we wanted to use during the processing. Our existing code is written in python, so we have choosen to process logs in two steps, logstash being the first.

Processing

As you may imagine, the log processor pulls the logs from the queue to processes them. Our log processor is a modular system that does a number of things with the logs, such as anayltics and performance monitoring. This post covers API usage tracking. Below is some pseudocode to illustrate the process.

It is worth explaining how the URL parameters are parsed. Each URL parameter needs to be canonized so that we don't count the same resource multiple times in the same hour. Not only do we have to deal with all sorts of URL ugliness, but we also must support redirected URLs, including link shortening services. Since our API has already processed the URL, the redirecting work is already complete, we just need to fetch it from our cache. First we canonize the URL, then we look in the cache to see if it redirects to any final URL, then we hash it.

Now that we have a customer ID, a rounded timestamp and a list of unique URL parameter hashes, it's finally time to track usage. The usage tracker is a simple integer that we increment as requests come in. URLs can only be incremented if the URL is unique for the hour, so another temporary data structure is needed to track hourly uniques. A Redis sorted set is used to track the hourly URLs. This allows us to not only keep track of how many unique URLs come in per hour, but also how many times each URL was requested. The sorted set's score is used to keep track of frequency. We can roll up this data when it is done being collected, and reuse it for analytics.

Triggers

There is a check to see if any interesting thresholds have been reached and if any actions need to be triggered as a result. When a customer reaches 80% and 100% of their monthly usage allowance, we send out an email notification letting you know. If the account is a free account and the 100% is reached, then the key is blocked until the end of the period.

Conclusion

If you made it this far, I'm proud of you. I hope this had shed some light on exactly how we track API usage. Thanks for reading!

Let's write about engineering shall we? I mean we do a lot of product work at Embedly and rarely do posts about the backend. Today I'm going to write about our battle against product versions within Chargify.

We have used Chargify since January to power our recurring payments. This was before Recurly and Stripe came out with their JavaScript libraries that make us very jealous. PCI compliance and Authorize.net are nightmares.

Last week we updated our pricing structures to reflect a $1 price decrease. One late night of QVC and an analysis of everyone else's pricing plans convinced us that $19 had a better conversion rate than $20 (So far this has held true for us as well).

Making this update in Chargify was a simple of matter of changing the product pricing within Chargify, and like magic people were signing up for $19 plans. The problem came when we wanted everyone to pay the same price.

What Chargify doesn't explain well is that every time you change a product you create a new version of it. This makes sense. If a customer signs up for a $10 plan and then you up that plan to $20 you usually want to grandfather that customer in. However, if you want all your customers on the most recent version of the product, there is no easy way to do that.

There is a thread on the Chargify support forum that explains what you must do to migrate the subscriptions to the new plan:

Unfortunately, there's no direct way to change existing subscriptions to the latest version - but you can create a temp product, switch them to that and then switch them back. That will make sure they are on the latest version.

So for every single user, we would have to switch them manually over to a temp plan and switch them back via the Chargify admin. For the hundreds of paying customers we have this would take hours. No. Thank. You.

Instead, I opted to use the Chargify API. What's incredible is that for a given subscription, Chargify does not pass back the version of the product that the user is on; only the most current one. I'll say that again. There is no way to know what version of a product the user is subscribed to via the API. Holler.

I created a small script that does all the heavy lifting. You have to first create a testing plan that is $0 with the handle 'testing'. After that you can run the following:

Done! You have now updated everyone to the correct plan. The first time I ran this script a few customers did not get updated. I don't know why, but we ended up having to spot check every subscription to make sure that it had the correct version. Annoying, but it only took us about 15 minutes with two people on board.

Hopefully this will save someone else some time when trying to migrate subscriptions to the most current plan.

It's time for your quarterly dose of design news with me, Tom, He Who Makes Things Pretty.

It's been a few months since we last redesigned the website, and I'm getting antsy. I would probably get throttled if I decided to completely redo the site for a 3rd time, so I have to settle for updating individual pages.

First in my queue is our pricing page. Our old pricing page was a simple, utilitarian table of information. It got the job done, but lacked pizazz. Feast your eyes on our new pricing page in all it's soft-gradient glory. For those of you who love tables, we still have it here.

(download)

Click here to download:

pricing-nip-tuck-rriJgEuuaEJrnBidfCkq.zip (1.19 MB)

When coming to our new and improved pricing page, the first thing you might say to yourself is "hey, these plans are all a dollar cheaper than I'm paying now." Good news! Your plan just got a dollar cheaper. You're welcome.

The second thing you might notice is the addition of a $99 plan. You talked, we listened. We know that making the jump from a $20 to a $200 (or $19 to $199) plan is a large pill to swallow. Our new Basic plan sits happily in the middle, ready to take your calls. The features are the same as the $19 Starter plan, you just get more URL requests per month.

In addition to the pricing changes, we have pushed our Enterprise-level features to the front. We understand that Enterprise customers have specific needs. Contact us today and we can discuss how Embedly can help your business succeed.

Thanks to Assistly for the design inspiration. Pricing pages aren't the sexiest things to design, and theirs was leaps and bounds better than all the others we sourced.

"This will save you a ton of time." - Everyone

jQuery Preview is an easy to use plugin that allows developers to create a URL submission tool using jQuery and Embedly. It looks something like this:

It's super easy to set up. All you need is a form, a little bit of Javascript and you can allow users to choose a thumbnail, edit metadata and embed content from over 200+ sources. It's f-ing amazing and to prove it here is all the code you need to implement it client side.

We put together a number of examples, so you should go check out the demo site. It has everything you need to get started.

Dive in, use it, it's awesome. For those interested, here is a bit more backstory.

Back in August we posted an EPIC, and I mean EPIC post on how to create a Facebook like URL submission tool with Embedly with EXT. It was a monster. 4,000+ words that was the authoritative masterpiece on creating an easy to use URL tool. It was so long that the average time on that page is almost 6 minutes, but guess what, no one used it.

I blame the fact that it wasn't packaged into a nice library and, well, it was written using EXT core. 

We used EXT because it was an example for a client, but this time around we decided to go with the most popular Javascript framework. We also mixed in a few tools that helped the process along. All the html is generated by the Mustache.js templating library and we use Underscore.js to add a bunch of utility functions.

Please let us know what you think. Here are a couple helpful links:

Code: https://github.com/embedly/jquery-preview

Documentation: http://embedly.github.com/jquery-preview/demo/

Issues: https://github.com/embedly/jquery-preview/issues

We have heard that users are having trouble embedding Vimeo.com videos in their Wordpress blogs. The issue has been highlighted as a redirect not happening from "www.vimeo.com" to "vimeo.com" as that is hardcoded in Wordpress's core code. The nice thing we did with our Embedly Wordpress plugin is made all the providers and endpoints accessible through an API call that can be updated occasionally when a provider goes down or makes changes to its endpoint.

**Update: Wordpress has updated the core code, not sure when that propogates to local wordpress instances. If you still want to use Embedly you will get about 150 more embed providers out of the plugin.

We know Vimeo embeds work in our Embedly Wordpress plugin and encourage you to try it out. You can search "embedly" in your plugins section to download it .  Also, here is a quick video of using the plugin with a Vimeo video:

As usual you can contact us at support@embed.ly with any issues. Enjoy!

In the coming weeks we're launching something brand new. If you've been on the fence about trying Embedly, because you don't have the time or the resources to spend on integration, then this is your lucky day.

We're introducing Embedly Anywhere, the easiest and fastest way to get up and running on Embedly. Anywhere is a drop-in solution. Just include our one line of JavaScript in your site, and we take care of the rest. It's fully customizable, and themeable.

We are looking for a few people to beta test this new offering for us. Sign up here, and we'll send you a message when we have Anywhere up and running.