On Friday, Embedly offered Hacker News a coding challenge. Apply.embed.ly asked developers to solve 3 different problems and submit their solutions. We didn't force people to apply for 1 of the 3 positions that we have open, just nerd out on some problems. We are going to talk about the results and the answers.

Here is a quick funnel of users to apply.embed.ly

1. Sum of Digits

Based on a question from Project Euler given the formula:

R(n) is the the sum of the digits for n!.
For example, 10! = 3628800
R(10) = 3 + 6 + 2 + 8 + 8 + 0 + 0 = 27.

We loved this question for the golf aspect. In python R(n) it can be written:

import math;
R = lambda x: sum(map(int, str(math.factorial(x))))

To actually solve it, almost everyone used a a brute force algorithm. Like so:

min([i for i in range(1000) if R(i) == 8001])

We got a total of 1008 distinct answers for this question. 758 were seen less than 2 times (some people tried to brute force the value)

The top three answers:

1. 787 (992)
2. 0 (384)
3. 802 (105)

2. Standard Deviation of P tags

This one was a mess, when the problem was first put up we had a very large and invalid, random-generated HTML file. If you used the Chrome console, lxml, nokogiri or ran the html through Tidy you got the 'correct' answer. If you used a sax parser, the answers were much different.

After a few confused tweets, we allowed any answer between 0.5 and 2.0. We then simplified the html greatly. This allowed people to manually count the depths of each p tag or use the white space to determine the depth. This may have defeated the purpose, but ok internet, you win.

We got a total of 242 distinct answers for this question. 117 were seen less than 2 times.

The top 3 answers:

1. 1.4 (335)
2. 0.767 (164)
3. 1.253 (101)

3. Zipf's Law.

We simplified Zipf's law to:

Z(x) = [x, x/2, x/3, x/4...]

This described the frequency distribution for words in a random body of text. Given that x = 2520 and a text of 900 unique words, how many words make up half the text?

This one got a little confusing too.

We can get the word count by using:

words = [2520/float(i) for i in range(1, 901)]
word_count = sum(words)

We can then iterate over the words till they are greater than 50% of the total word count.

min([i for i in range(30) if sum(words[:i]) > sum(words)/2.0])

It got a bit hairy when it came to rounding. We were in the wrong here by using float instead of integer because it doesn't make sense to have fractional word counts. We should have accepted 21 instead of 22.

The top 3 answers:

1. 22 (450)
2. 21 (204)
3. 20 (120)

Hacking

We intentionally made it easy to hack apply.embed.ly. The url paths were /1 /2 /3 and every time you got a question right, we just added a cookie 'au_embedly_1=true' for the problem you solved. Only Will Pearson used this to his advantage and skipped a problem.

Standing Out.

Some notable examples of different ways people solved this.

1. A couple people solved it in the Chrome console, no text editor needed.
2. 6 minutes. The total time it took one college sophomore to solve it.
3. Ruby one liners for all: https://gist.github.com/1792968/cbb3f5c22ff2e7d174734c780df87e8b9e85153e
4. All in Mathematica: https://gist.github.com/1797321
5. You can solve the first problem in J in 27 chars: "(+/ "1 f"0 !i.1000x) i.8001"
6. A number of people used excel to solve a majority of the problems. There seems to be a lot of finance nerds lurking on HN.

Gists:

If you are interested in seeing the solutions everyone posted here you go. I embedded a gist of gists because I cannot for the life of me figure out how to get Posterous not to embed them.

https://gist.github.com/1820286

Permalink | Leave a comment  »

We woke up yesterday with smiles on our faces and using our favorite chrome plugin (Parrotfish) . Then we got the news that the #new#new Twitter was released. Jaws dropped, tweets flew in, and profanities flew out. Our users expected results. @hotdogsladies tweeted that lovemaking was just not the same without Embedly. We concur.

For a brief moment we considered retiring Parrotfish. Surely in this latest release Twitter would have implemented embeds the way they should have from the beginning. Lucky for us, it appears to be the same crippled system that caused us to create Parrotfish in the first place.

So, off to the Batcave Sean and Bob went. Afterall, we do our best work with bats circling. Who doesn't? They (Sean and Bob, not the bats) woke up this morning, ready again to tread through the depths of a Twitter re-design, this time armed with some new toys that we have created over the last few months.

We now present to you the latest and greatest Parrotfish ready to conquer your timeline (the Twitter one, not the Facebook one):

•  Enabled with SSL support for embeds and images. (Secure)
•  Better favicons and logos.
• Available in Chrome and Safari. (FF you're next)

Get it right away at Embedly Labs.

Permalink | Leave a comment  »

Last week we took a break from bug fixing, redesigning, and development. We held our own internal Hack Week: 4 developers, 4 completely different projects, all using or enhancing the Embedly service.

Arthur spent the week adding more social features to Embedly. We want to be able to answer the question: "what's the most popular content on my site?" Arthur developed a Reddit-like voting system for embeds, that get tallied by us and displayed with the rest of our Analytics.

Bob created a web socket proxy for the Embedly API, developed using node.js, because Bob loves node.js. The proxy allows for truly asynchronous requests to the Embedly API, returning embeds as they finish instead of all at once. If anyone is interested, Bob will add documentation when he has some free time, probably during the next Embedly Hack Week.

Sean, the master of Chrome plugins, developed a super top secret Chrome plugin. We could tell you about it, but then we'd have to kill you. Or we could make you sign an NDA, but paperwork is messy.

We sometimes get too focused on business development, answering support tickets, and making sure the servers stay up and running. It's nice every once in a while to take a step back and reap what we've sown. We're constantly surprised with what we've managed to accomplish over the last two years.

We love to hear how others are using the Embedly API. Let us know in comments, and as always, we're available at support@embed.ly with any questions.

Permalink | Leave a comment  »

Permalink | Leave a comment  »

In our opinion censorship on the web is never a good thing. While we are not here to get on the soapbox or try to instill values, SOPA is just bad for business. 

Embedly is a company that deals with millions of links a day. If you start censoring what people can link to, inevitably that hurts us. 

When a user embeds an infringing video, who is at risk? The site that embedded the video, the video hosting service or the delivery mechanism? I certainly don't want to find out. Maybe it will be like prohibition and Embedly will become the bootleggers of our time, transporting content from provider to publisher.

If you visit embed.ly today you will see a tiny band over our logo in solidarity for the cause.

We encourage you to read the bill, watch the video and visit americancensorship.org to learn more.

Sean

Permalink | Leave a comment  »

In this post we will take an in depth look at how Embedly tracks API usage. Warning: this post is not for the faint of heart.

Overview

Our API consists of a cluster of tornado boxes behind two nginx load balancers. API usage is based on the nginx access log at the load balancer. This allows us to seperate tracking from the API.

Nginx logs are transported from the load balancer to the log queue. The log queue is simply a Redis list. Logs are transported to the queue with logstash. Logstash also parses the access logs into structured data, making it easier to process them later. We could have written all of our log processing as a logstash filter, but we have a lot of existing code that we wanted to use during the processing. Our existing code is written in python, so we have choosen to process logs in two steps, logstash being the first.

Processing

As you may imagine, the log processor pulls the logs from the queue to processes them. Our log processor is a modular system that does a number of things with the logs, such as anayltics and performance monitoring. This post covers API usage tracking. Below is some pseudocode to illustrate the process.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

def process(log):
      api_key, urls, status = parse(log)

      if status in (200,404):

          rounded_timestamp = round(log.timestamp)
          url_ids = hash_urls(urls)
          customer_id = get_customer_id(api_key)

          zkey = "hourly::$customer_id::$rounded_timestamp"
          redis.multi()
          # count unique URLs for hour before adding new set
          start = redis.zcard(zkey)
          for url_id in url_ids:
              redis.zincrby(zkey, url_id, 1)
          # count unique URLs for the hour after adding new set
          end = redis.zcard(zkey)
          redis.exec()

          new_url_count = end - start

          pkey = "usage::$customer_id::$period_end"
          new_period_count = redis.incrby(pkey, new_url_count)
          old_period_count = new_period_count - new_url_count
          for threshold in thresholds:
              if new_period_count >= threshold and \
                      old_period_count < threshold:
                  threshold.action(customer_id)

It is worth explaining how the URL parameters are parsed. Each URL parameter needs to be canonized so that we don't count the same resource multiple times in the same hour. Not only do we have to deal with all sorts of URL ugliness, but we also must support redirected URLs, including link shortening services. Since our API has already processed the URL, the redirecting work is already complete, we just need to fetch it from our cache. First we canonize the URL, then we look in the cache to see if it redirects to any final URL, then we hash it.

Now that we have a customer ID, a rounded timestamp and a list of unique URL parameter hashes, it's finally time to track usage. The usage tracker is a simple integer that we increment as requests come in. URLs can only be incremented if the URL is unique for the hour, so another temporary data structure is needed to track hourly uniques. A Redis sorted set is used to track the hourly URLs. This allows us to not only keep track of how many unique URLs come in per hour, but also how many times each URL was requested. The sorted set's score is used to keep track of frequency. We can roll up this data when it is done being collected, and reuse it for analytics.

Triggers

There is a check to see if any interesting thresholds have been reached and if any actions need to be triggered as a result. When a customer reaches 80% and 100% of their monthly usage allowance, we send out an email notification letting you know. If the account is a free account and the 100% is reached, then the key is blocked until the end of the period.

Conclusion

If you made it this far, I'm proud of you. I hope this had shed some light on exactly how we track API usage. Thanks for reading!

Permalink | Leave a comment  »

Let's write about engineering shall we? I mean we do a lot of product work at Embedly and rarely do posts about the backend. Today I'm going to write about our battle against product versions within Chargify.

We have used Chargify since January to power our recurring payments. This was before Recurly and Stripe came out with their JavaScript libraries that make us very jealous. PCI compliance and Authorize.net are nightmares.

Last week we updated our pricing structures to reflect a $1 price decrease. One late night of QVC and an analysis of everyone else's pricing plans convinced us that $19 had a better conversion rate than $20 (So far this has held true for us as well).

Making this update in Chargify was a simple of matter of changing the product pricing within Chargify, and like magic people were signing up for $19 plans. The problem came when we wanted everyone to pay the same price.

What Chargify doesn't explain well is that every time you change a product you create a new version of it. This makes sense. If a customer signs up for a $10 plan and then you up that plan to $20 you usually want to grandfather that customer in. However, if you want all your customers on the most recent version of the product, there is no easy way to do that.

There is a thread on the Chargify support forum that explains what you must do to migrate the subscriptions to the new plan:

Unfortunately, there's no direct way to change existing subscriptions to the latest version - but you can create a temp product, switch them to that and then switch them back. That will make sure they are on the latest version.

So for every single user, we would have to switch them manually over to a temp plan and switch them back via the Chargify admin. For the hundreds of paying customers we have this would take hours. No. Thank. You.

Instead, I opted to use the Chargify API. What's incredible is that for a given subscription, Chargify does not pass back the version of the product that the user is on; only the most current one. I'll say that again. There is no way to know what version of a product the user is subscribed to via the API. Holler.

I created a small script that does all the heavy lifting. You have to first create a testing plan that is $0 with the handle 'testing'. After that you can run the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

"""
A simple script to update Chargify subscriptions to the most recent version.
"""
import base64
import requests
import json

API_KEY = 'Chargify API Key'
HOST = 'https://<Chargify Host Name>.chargify.com'

headers = {
    "Authorization": "Basic %s" % base64.encodestring('%s:%s' % (API_KEY, 'x'))[:-1],
    "User-Agent": "Migrate-to-Latest-Version",
    "Content-Type": "application/json"
}

def update_plan(id, handle):
    
    print 'Updating %s to %s' % (id, handle)

    d = {
        "subscription":{
            'product_handle' : handle
        }
    }    
    data = json.dumps(d)
    url = '%s/subscriptions/%s.json' %  (HOST, id)

    r = requests.put(url, data=data, headers=headers)
    
    if r.error:
        print 'Error: %s ID: %s HANDLE: %s' % (r.error, id, handle)


def update(pages=10):
    
    for p in range(1, pages+1):
        r = requests.get('%s/subscriptions.json?per_page=200&page=%s' % (HOST, p), headers=headers)
        data = json.loads(r.content)

        # No more subs:
        if not data:
            break

        # We only update the plans that were not canceled.
        subs = [s['subscription'] for s in data if s['subscription']['canceled_at'] is None]

        for s in subs:
            handle = s['product']['handle']
            id = s['id']
    
            #update to testing account.
            update_plan(id, 'testing')
    
            #update back to the right plan
            update_plan(id, handle)


#Spot checking and testing.
def update_one(id):
    sub = get_id(id)
    handle = sub['product']['handle']
        
    #update to testing account.
    update_plan(id, 'testing')
    
    #update back to the right plan
    update_plan(id, handle)

#Utils that help 
def get_id(id):
    url = '%s/subscriptions/%s.json' % (HOST, id)
    r = requests.get(url,  headers=headers)
    return json.loads(r.content)['subscription']
    
def get_all(pages=10):
    
    subs = []
    for p in range(1, pages+1):
        r = requests.get('%s/subscriptions.json?per_page=200&page=%s' % (HOST, p), headers=headers)
        data = json.loads(r.content)
        
        # No more subs:
        if not data:
            break

        subs.extend([s['subscription'] for s in data])

    return subs

if __name__ == '__main__':
    update()

Done! You have now updated everyone to the correct plan. The first time I ran this script a few customers did not get updated. I don't know why, but we ended up having to spot check every subscription to make sure that it had the correct version. Annoying, but it only took us about 15 minutes with two people on board.

Hopefully this will save someone else some time when trying to migrate subscriptions to the most current plan.

Permalink | Leave a comment  »

It's time for your quarterly dose of design news with me, Tom, He Who Makes Things Pretty.

It's been a few months since we last redesigned the website, and I'm getting antsy. I would probably get throttled if I decided to completely redo the site for a 3rd time, so I have to settle for updating individual pages.

First in my queue is our pricing page. Our old pricing page was a simple, utilitarian table of information. It got the job done, but lacked pizazz. Feast your eyes on our new pricing page in all it's soft-gradient glory. For those of you who love tables, we still have it here.

See the full gallery on Posterous

When coming to our new and improved pricing page, the first thing you might say to yourself is "hey, these plans are all a dollar cheaper than I'm paying now." Good news! Your plan just got a dollar cheaper. You're welcome.

The second thing you might notice is the addition of a $99 plan. You talked, we listened. We know that making the jump from a $20 to a $200 (or $19 to $199) plan is a large pill to swallow. Our new Basic plan sits happily in the middle, ready to take your calls. The features are the same as the $19 Starter plan, you just get more URL requests per month.

In addition to the pricing changes, we have pushed our Enterprise-level features to the front. We understand that Enterprise customers have specific needs. Contact us today and we can discuss how Embedly can help your business succeed.

Thanks to Assistly for the design inspiration. Pricing pages aren't the sexiest things to design, and theirs was leaps and bounds better than all the others we sourced.

Permalink | Leave a comment  »

"This will save you a ton of time." - Everyone

jQuery Preview is an easy to use plugin that allows developers to create a URL submission tool using jQuery and Embedly. It looks something like this:

It's super easy to set up. All you need is a form, a little bit of Javascript and you can allow users to choose a thumbnail, edit metadata and embed content from over 200+ sources. It's f-ing amazing and to prove it here is all the code you need to implement it client side.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<html>
  <head>
    <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js" type="text/javascript"></script>
    <script src="http://scripts.embed.ly/p/0.1/jquery.preview.full.min.js" type="text/javascript"></script>
    <link rel="stylesheet" href="http://scripts.embed.ly/p/0.1/css/preview.css" />
  </head>
  <body>
    <form action="/update" method="POST">
        <input id="url" type="text" name="url"/>
    </form>
    <script>
        $('#url').preview({key:'your_embedly_key'})
    </script>
  </body>
</html>

We put together a number of examples, so you should go check out the demo site. It has everything you need to get started.

Dive in, use it, it's awesome. For those interested, here is a bit more backstory.

Back in August we posted an EPIC, and I mean EPIC post on how to create a Facebook like URL submission tool with Embedly with EXT. It was a monster. 4,000+ words that was the authoritative masterpiece on creating an easy to use URL tool. It was so long that the average time on that page is almost 6 minutes, but guess what, no one used it.

I blame the fact that it wasn't packaged into a nice library and, well, it was written using EXT core. 

We used EXT because it was an example for a client, but this time around we decided to go with the most popular Javascript framework. We also mixed in a few tools that helped the process along. All the html is generated by the Mustache.js templating library and we use Underscore.js to add a bunch of utility functions.

Please let us know what you think. Here are a couple helpful links:

Code: https://github.com/embedly/jquery-preview

Documentation: http://embedly.github.com/jquery-preview/demo/

Issues: https://github.com/embedly/jquery-preview/issues

Permalink | Leave a comment  »

We have heard that users are having trouble embedding Vimeo.com videos in their Wordpress blogs. The issue has been highlighted as a redirect not happening from "www.vimeo.com" to "vimeo.com" as that is hardcoded in Wordpress's core code. The nice thing we did with our Embedly Wordpress plugin is made all the providers and endpoints accessible through an API call that can be updated occasionally when a provider goes down or makes changes to its endpoint.

**Update: Wordpress has updated the core code, not sure when that propogates to local wordpress instances. If you still want to use Embedly you will get about 150 more embed providers out of the plugin.

We know Vimeo embeds work in our Embedly Wordpress plugin and encourage you to try it out. You can search "embedly" in your plugins section to download it .  Also, here is a quick video of using the plugin with a Vimeo video:

As usual you can contact us at support@embed.ly with any issues. Enjoy!

Permalink | Leave a comment  »

In the coming weeks we're launching something brand new. If you've been on the fence about trying Embedly, because you don't have the time or the resources to spend on integration, then this is your lucky day.

We're introducing Embedly Anywhere, the easiest and fastest way to get up and running on Embedly. Anywhere is a drop-in solution. Just include our one line of JavaScript in your site, and we take care of the rest. It's fully customizable, and themeable.

We are looking for a few people to beta test this new offering for us. Sign up here, and we'll send you a message when we have Anywhere up and running.

Permalink | Leave a comment  »

We wanted to share a quick feature update with developers. Embedly now lets you whitelist IP addresses through your Dashboard!

About a month ago we launched a feature that allowed developers to whitelist certain referrers to protect their API key from being used maliciously. Today we are going to let server side implementations of Embedly do the same thing. Just head to the Dashboard and you will see a new box on your homepage `Manage your IPs`.

If you click `Manage` you will be taken to the IP Address management view that allows developers to input any IP Address they want to whitelist. These changes are instantaneous, so be careful! We don't want you bringing down production implementations, now do we?

As with Referrers, we also allow you to test out your patterns through a handy pattern checker tool. It makes it simple to add patterns andd then verify that your IP Addresses will work.

That's it! Now get whitelisting. If you have any questions please contact us at support@embed.ly.

Permalink | Leave a comment  »

It's been a little over a month since we simplified Embedly into one API. It went very well and we learned a lot in the process. The most common point of confusion was over how Embedly counts URLs. In the past we counted every URL requested of us against your plans, which forced many of you to implement serious caching schemes on your end to keep costs down. That's annoying, we know, so we are going to change it and we think you're going to like it.

First and foremost this will not adversely affect anyone. Everyone's overall totals will be going down. Before we go into the new system, it's helpful to understand how we used to count URLs to know what is changing.

Account Metering

Before, Embedly counted each and every URL you sent to Embedly towards your monthly count. So if you used Embedly client side, and had 10 URLs on a page each time that page was viewed your count would go up by 10. This means that with only 1,000 page views in a day, a developer could reach their monthly free plan limit in one day.

New plan: We only count unique URLs requested per hour, per month.

Unique URLs per Hour per Month

Lets look at the same 10 URLs from before. Between 1pm and 2pm, you get 1000 people visiting your site. The 10 URLs are requested from Embedly 1000 times, but each one is only counted once against your plan.

At 2pm our counters reset. Between 2 and 3pm the same 10 URLs are requested another 1000 times on your site. We, again, count each one once against your plan. So instead of 20,000 requests from 1pm - 3pm, we count 20 requests.

Put another way, each unique URL you send Embedly will be counted at most 24 times per day against your plan.

More Good News

As of today everyone will be getting analytics for the current day. This will allow you to see the urls that you are requesting. If you want to go back in time the analytics package will be available at the core package.

We have created a landing page with all this information so you can learn more about it. http://embed.ly/pricing.

Please feel free to contact us with any questions or comments at support@embed.ly.

Thanks for being a part of the Embedly Platform.

Permalink | Leave a comment  »

Permalink | Leave a comment  »

One of our clients is currently using Ext and Embedly to implement a Facebook-like link submission tool. While writing up some examples we went overboard and ended up coding the whole thing. It's worth sharing with the community as a whole, because we have seen this question many times before. This post ended up being a lot longer than expected, some 4,000+ words describing the process and design decisions that go into letting users submit links. Herein lies the bible of using Embedly to create a URL submission tool with Ext Core.

First off, note that I am not an Ext expert. In fact this is the first I have ever used the tool. While I disagree with its obsession with ids, I like how Ext creates HTML and the fly method. If you see anything that is off or just looks wrong, blame ignorance.

The final demo is hosted at embedly.github.com/embedly-tutorial-ext. You can submit a URL and see how all the interactions work. The demo is optimized for modern browsers and relies heavily on localStorage to display the Feed, so older browsers will fail hard. The source code is available at github.com/embedly/embedly-tutorial-ext. We encourage you to clone it and follow along.

Overview

What we are trying to build here is a fairly common design pattern that allows users to preview the metadata for a link before submitting it to their feed. If you have used Facebook, Google+, Yammer or any number of services you have seen this before. Below are a few screenshots from various different services.

Facebook

Yammer (Uses Embedly)

Google+

The interface for all three are fairly similar. Each has a textarea for the status and when a user enters a URL, they all recognize it and fetch the metadata associated with the URL. The user can then select a thumbnail and edit the title and description. The question is why? Well it turns out that humans are much better at picking thumbnails, titles and descriptions than algorithms will ever be. It's worth delving into an example and showing where all the metadata comes from. We will use the 10k Apart website as an example and this is a screenshot of the homepage at the time of writing.

Let's look at the metadata in the Facebook, Yammer, Google+ comparison from above. The title is the same for each, but the first image and description are different. Why is that? Each service uses a different set of algorithms to pull data out of the page. The 10k Apart site doesn't specify any image_src, og:image, decription or og:description tags in the head of the html so it's everyman for themselves . Pulling the best image or a great excerpt is hard and when we let code handle it there is bound to be variance and incorrect data.

When it comes down to it though, it should be up to the user. Any of the three images above could work, but at the end of the day a user, not code, will always choose the best image. So the task is simple, create a form that allows a user to input a URL and select the thumbnail, title and description they wish to show. There are however a bunch of other considerations that you will invariable end up coming across that we will look at as well.

Preview Endpoint

One of Embedly's API endpoints was specifically made for this task. The preview endpoint expands on oEmbed and passes back an array of images, content, place information, event details as well as the normal oEmbed html data. It holds a lot of information, but we are only going to use the following attributes. I'm going to go into detail here because I think it's import that you know where the data is coming from and how to use it.

type

The type of the response that we got back from the server. 90% of the time you will get back a type of html, but another very common one is image. There is a whole list of response types, but generally most people code for html and image and give up on the other types. Later we will go through what that actually means.

title

Title of the page. Embedly looks for a title in the API response, the Open Graph title tag or the <title> tag in the head of the doc in that order . For 10k.aneventapart.com we found the title tag:

description

A description or brief excerpt from the page. Embedly looks for a title in the API response, the Open Graph description, the <meta name="description"> tag in the head of the doc in that order. If none of those exist, or are too short, Embedly will use an algorithm to try to find a good excerpt from the page. It does this by looking for common clues like a string of p tags, divs with lots of text at a similar depths and a whole slue of other factors.

Going back to the Facebook, Google+ and Yammer examples above here are the descriptions that each give back

Facebook:

''

Google +:

10k Apart Responsive Edition. Inspire the Web with Just 10k. Read the Rules Submit an Entry. The Gallery. 1-1 of 1. Launch Details. Colorrrs. Colorrrs. Dave Rupert. Enter Now. Enter Now. The Rules (FA...

Yammer (Embedly):

Total file size including images, scripts & markup can't be over 10k zipped. Details Use the approved list of libraries without it counting against your 10K. Details We encourage HTML5, and apps must work equally well in IE10 PP2, Firefox & a Webkit browser. Details Applications need to be responsive.

As you can see Facebook gave up and was unable to pull a description. Google found the best first line, but quickly degrades to a bunch of nonsense. Embedly found the largest body of text and tried to use that. It's the longest and respects sentence structure, but the description of the rules, not of the contest itself. The fun thing is, if you visit the page, none of this text is initially viewable to the user. It's in divs that are hidden by css, and therefore we have no way of knowing if they are displayed or not.

A user could easily intervene here and edit the description to something that made a little more sense. As a side note, it's very easy to pull text out of a page, it's hard to pull a good excerpt and it's even harder to know when you should be displaying one at all.

images

This is a JSON array of images of possible thumbnails for the user to select. For 10k.aneventapart.com, the JSON array looks like this:

[
  {
    "url": "http://10k.aneventapart.com/Uploads/501/Thumbnail1.jpg",
    "width": 600,
    "height": 400
  },
  {
    "url": "http://10k.aneventapart.com/Content/img/enter_now.jpg",
    "width": 600,
    "height": 400
  },
  {
    "url": "http://10k.aneventapart.com/Content/img/10k_logo.png",
    "width": 235,
    "height": 144
  }
]

As you can see Embedly values the larger images in the middle of the page greater than the smaller logo at the top of the page. Within the middle of the page we value images that appear higher in the page.

Images are pulled from a bunch of different sources: API responses, the Open Graph image tag, the image_src link tag and the page itself. Embedly follows all these images to get the correct height, width and verify that they exist. Scoring these images is really complicated. Each image is scored based on where they lie in the page, what the image type is, if they matched a list of commonly used ad servers, did the image redirect and a whole slew of other factors that have been added over time.

Still with all these factors, it's hard to pick the right image every time.

provider_display

provider_display is different from oEmbed's provider_name. It is a very easy way to get the domain of the provider. For example, http://www.bbc.co.uk/news/science-environment-14391929 has a provider_display` of www.bbc.co.uk. This allows you to show a user what domain they will be visiting.

provider_url

provider_url works in conjunction with provider_display. It's the URL of the provider. Most of the time you can use it to link to the provider like so:

object

object is fairly similar to an oEmbed object, but striped down. The idea here is that there is an object associated with the url that you passed to Embedly. There are three types: photo, video and rich . video and rich can be treated the same code wise when displaying the embed. The html element can just be set to the innerHTML of the feed item. Here is a simple example in js:

if (preview.object.type in {'video':'', 'rich':''}){
  Ext.fly('item').dom.innerHtml = preview.object.html;
}

The photo is a little different in that it there is no html attribute, but a URL instead. You can very easily use it to build the html though:

if (preview.object.type == 'image'){
   Ext.fly('item').dom.innerHtml = '<img src="'+preview.object.url'"/>';
}

Note that we are not using the width and height attributes on the img tag. The images that are passed back are all different sizes so, it's best to use the css style max-width like so:

#item img {
    max-width:400px;
}

Don't bother messing with the height. People know how to scroll, so designs that are tolerant to different heights of images are the best.

Retrieval

There are two sections to the feed; retrieval and display. Retrieval is the long section that describes grabbing metadata from Embedly and allowing the user to edit it before submission. Display is much shorter and just goes into tips and tricks for displaying the data.

We start off with the following simple form:

And the base for our Preview obj that we will use to wire up all the supporting functions. You can use any of the 20 different object declaration patterns in JavaScript, ours just happens to look like this:

var Preview = (function(){
  var Preview = {};
  return Preview;
})();

A user will come to your site in hopes of posting a status of some sort and this status may contain a link. There are a few events that we need to listen to here in order to create the desired effect: paste, blur and keyup.

paste

Easily the most common way that users move links around. The event fires after anything is pasted into the object you are listening on. In Ext you can listen to the event like so:

Ext.EventManager.on("id_status", 'paste', Preview.fetchMetadata);

The paste event is a little inconsistent however and at least in Chrome actually fires before the textarea is filled. Because of that it's better to set a short timeout to make sure the pasted value is there:

Ext.EventManager.on("id_status", 'paste', function(){
    setTimeout(Preview.fetchMetadata, 250);
});

blur

When the the status textarea loses focus we need to check if the user added anything to it. While keyup and paste will catch 95% of the cases this one is nice to have:

Ext.EventManager.on("id_status", 'blur', Preview.fetchMetadata);

keyup

If a user wants to be so bold that they actually type in the URL, we want to fetch it as soon as they hit the spacebar. This one is a little more tricky because if they are manually typing a url they may edit it a few times causing repeat calls. While we are not going to worry about that here it's just something to think about.

Ext.EventManager.on("id_status", 'blur', Preview.onKeyUp);

The onKeyUp function has a different set of rules than just fetchMetadata as we have to listen for just the spacebar after a URL has been entered:

onKeyUp : function(e,t){
  // Ignore Everything but the spacebar Key event.
  if (e.getKey() != 32) return null;

  //See if there is a url in the status textarea
  var url = Preview.getStatusUrl();
  if (url == null) return null;

  // If there is a url, then we need to unbind the event so it doesn't
  // fire again. This is very common for all status updaters as
  // otherwise it would create a ton of unwanted requests.
  Ext.EventManager.un("id_status", 'keyup', Preview.onKeyUp);

  //Fire the fetch metadata function
  Preview.fetchMetadata();
},

The unbind is very important here. A user may go back and edit the URL a hundred times here. We assume they got it right the first time, otherwise we will update the URL when the textarea loses focus.

Now that all the events are hooked up we need to pull the URL out of the status textarea. While we won't be handing multiple urls, it's fairly easy to pull out a single one:

var status = Ext.fly('id_status').getValue();

//Simple regex to make sure the url is valid.
var urlexp = /http(s?):\/\/(\w+:{0,1}\w*)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%@!\-\/]))?/;

//Match the status against the urlexp
var matches = status.match(urlexp);

return matches? matches[0] : null

This will catch any url as long as the user has entered the http or https scheme. As we know the scheme is becoming less and less prevalent and most users expect it to work if they leave out the http://. For example it shouldn't matter if a user enters nyti.ms/qdGs9A or http://nyti.ms/qdGs9A. You could be clever here and just update the original regex, but I'm not, so I will create a new one.

var urlexp = /[-w]+(.[a-z]{2,})+(S+)?(/|/[w#!:.?+=&%@!-/])?/g;

var matches = status.match(urlexp);

return matches? 'http://'+matches[0] : null

This regex is going to catch a number of false positives here. Users editing their statuses may type something like "I love it.seriously ..." which will trigger a request. You could do something clever with publicsuffix.org or just be better with regexes. What's interesting to note is that neither Facebook or Google Plus offer this feature. They both make you use the 'link' function in order to enter a URL without a scheme. They must know something or Facebook set the trend and Google+ just copied.

Once you actually have the URL from the status textarea we have to make a JSONP request to the Embedly Preview endpoint to get the metadata associated with that URL. I used jsonp.js that was bundled in examples/jsonp in the Ext Core download. Here is the code to get it done, then we will go into all the available options:

// Sets up the parameters we are going to use in the request.
params = {
  url:url,
  key:':key', // replace with your key.
  autoplay:true,
  maxwidth:500,
  wmode : 'opaque',
  words : 30
}

// Make the request to Embedly. Note we are using the preview endpoint:
// http://embed.ly/docs/endpoints/1/preview
Ext.ux.JSONP.request('http://api.embed.ly/1/preview', {
  callbackKey: 'callback',
  params: params,
  callback: Preview.metadataCallback
});

When setting up the parameters you have a number of options. We are going to go into detail on a number of them here so you know just how each will effect your application.

url

The url that you want to retrieve metadata for. Ext takes care of encoding the URL, but if you aren't using a library you need to escape the URL. Something like this works:

var url = encodeURIComponent('http://embed.ly')

key

Your Embedly API key. You can sign up for one at embed.ly/pricing. This tutorial uses the Preview endpoint which is only available at the "Starter" plan level and above.

maxwidth

This is the maximum width of the embed in pixels. maxwidth is used for scaling down embeds so they fit into a certain width. If the container for an embed is 500px you should pass { maxwidth: 500 } in the parameters. For example, if you don’t set a maxwidth for the a Vimeo video Embedly will return the following html:

This width may cause the embed to overflow the containing div. If we pass { maxwidth: 500 } the html will be:

It is highly recommended that developers pass a maxwidth to Embedly.

width

width will scale embeds type rich and video to the exact width that a developer specifies in pixels. Embeds smaller than this width will be scaled up and embeds larger than this width will be scaled down. During the scaling process the embed may become distorted, so if you can, it's best to use the maxwidth parameter.

Width is however really useful if you are working with a small set of providers that you know scale really well. It will scale up embeds to give a nice constant feel of every embed in your application.

wmode

wmode will append the wmode value to the flash object. Possible values include window, opaque and transparent. Generally you always want to have { wmode : 'opaque' } in the parameters. This prevents embeds from being rendered on top of modals or other html positioned on top of them.

autoplay

Tells video embeds to start playing as soon as they are loaded. Generally this is a reallyannoying feature of some sites, but in our case it's a great feature. It allows us to start playing the video as soon as the the user clicks on the thumbnail.

words

The words parameter has a default value of 50 and works by trying to split the description at the closest sentence to that word count. For example, the following lorem ipsum description is made up of 33 words and 5 sentences:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus dapibus auctor aliquam. Donec vitae justo ligula, id luctus ligula. Duis eget mauris lacinia sapien aliquet vulputate a et orci. Sed eu imperdiet sem.

Now by default, Embedly will return all 33 words, but say you want only 20 words. By passing { words : 20} Embedly would return:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus dapibus auctor aliquam. Donec vitae justo ligula, id luctus ligula.

This is actually only 19 words, but we split at the closest sentence. Words is really useful for controlling how long the descriptive text for each URL is. In this case we are going to use 30 words to not overwhelm the page with text.

chars

chars is like words, but instead of truncating to the nearest sentence, Embedly will blindly truncate a description to the number of characters you specify adding ... at the end when needed. For the above description, if we set { chars : 100 } it will return:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus
dapibus auctor aliquam. Donec ...

Display

Once you make the request we have to deal with the data that we get back from Embedly. We discussed the different parts of the object that we are going to use earlier, now it's just putting together the pieces. A lot of this is actually up to the individual developer, but here are some tips an tricks. We declare the metadataCallback from before as:

metadataCallback : function(obj){
  // Deal with the object here.
}

The first thing you need to do is validate the request. Every obj should have a type. If it's not there this is a clear sign that something is off. This is a basic check to make sure we should proceed. Generally will never happen, but it's a nice to have just in case:

if (!obj.hasOwnProperty('type')){
    console.log('Embedly returned an invalid response');
    return false;
}

The next thing you need to make sure that there isn't an error. If Embedly is sent an invalid URL, the URL returns a 404 or some other error Embedly will return an object of type error. In this general case the default workflow should occur. Generally you need not alert the user, just proceed as everything is happening normally:

if (obj.type == 'error'){
    console.log('URL ('+obj.url+') returned an error: '+ obj.error_message);
    return false;
}

At this point you have a response that you can work with, but you need to filter out types that you do not want to handle. In this case we will only be handling html and image type responses. Others link pdf of video we can build in another day:

if (!(obj.type in {'html':'', 'image':''})){
    console.log('URL ('+obj.url+') returned a type ('+obj.type+') not handled');
    return false;
}

To wire up the form to work on POST we need to set all the attributes to hidden inputs within the form. When the user is done and hits submit it will send all this data back to the server for saving. To do this we iterate over a list of elements we want to save:

Ext.each(Preview.attrs, function(n){
  Ext.DomHelper.append('preview_form', {
    tag:'input',
    name : n,
    type : 'hidden',
    id : 'id_'+n,
    value : obj.hasOwnProperty(n) && obj[n] ? encodeURIComponent(obj[n]): ''
  });
});

You can set Preview.attrs to pretty much anything you want, but in our case we use:

attrs : ['type', 'original_url', 'url', 'title', 'description',
    'favicon_url', 'provider_url', 'provider_display', 'safe', 'html', 'thumbnail_url'],

The last part of the metadataCallback function is handing off the obj to be rendered by a Display object. The Display object lets us change how the link preview is displayed without worrying about how it effects the Preview object. It also helped us create multiple versions of the demo:

Preview.Display.render(obj);

Rendering the link form is actually pretty boring. You show read the code , but at the end of the day, it's going to be up to you. The only thing to remember is to to update the hidden inputs with the correct values after the user has changed any data. For example we run this after a user has updated the title:

Ext.fly('id_title').dom.value = encodeURIComponent(elem.dom.value);

Now it's all about saving the data. You can do it as a basic post, but why make the user wait around for the save to happen? Instead we can write the status to the feed and save it asynchronously. This way, to the user, it appears as though the save happened instantaneously. First we need to bind a callback to the submit handler of the form.

Ext.EventManager.on("preview_form", "submit", Preview.Feed.submitFeedItem);

The Feed object is like Display in that we can switch in and out different implementations for various effects. The basic Feed object holds the CRUD functions for a set of data:

var Feed = {
  createFeedItem : function (data){}},
  storeFeedItem: function(data){},
  populateFeed: function(){},
  submitFeedItem: function(e,t){},
  deleteFeedItem: function(e,t){}
}

The submitFeedItem call back need to pull out all the data out of the form like so:

var data = {};
// Get the data we need out of the form.
Ext.select('#preview_form input').each(function(e){
  data[e.dom.name] = decodeURIComponent(e.dom.value)
});

//Create the Feed Item and display it in the feed.
Preview.Feed.createFeedItem(data);

//Save the Feed Item
Preview.Feed.storeFeedItem(data);

Note that when we grab the data out of the form we need to decode it via the decodeURIComponent function. Once that data is out of the form, we can then use it to create a feed item on the fly and then save it. The basic structure of a feed item for us looks like so:

We build that via a giant JSON object that you can see here . The important thing here is to also save the item data into div as data properties. The HTML5 spec describes a method for saving off custom data attributes that we will use here. To add these attributes the the outer div .item we can use something like this when building the JSON object:

Ext.each(Preview.attrs, function(n){
  elem['data-'+(n == 'html' ? 'embed' : n)] = encodeURIComponent(data[n])
});

We change the data-html to data-embed because it appears to be reserved by Ext or the browser, but I didn't investigate to deeply. Once this is in place we can get the title for any item like so:

elem.dom.dataset.title;

To be on the safe side of browser bugs we still use:

elem.dom.getAttribute('data-title')

Using these data attributes we can create an event to autoplay the video when a user clicks the thumbnail. In order the accomplish this we need to know that the url has a video associated with it. In the metadataCallback from above we actually change the type of the embed after we do a number of the checks to video or rich instead of html. We do this by updating the hidden inputs to have the correct values:

if (obj.object && obj.object.type in {'video':'', 'rich':''}){
  Ext.fly('id_html').dom.value = obj.object.html;
  Ext.fly('id_type').dom.value = obj.object.type;
}

If the type is video or rich we change the the thumbnail html to look like so:

This creates an embed that looks like

We can then use Ext to bind the click event to the Feed.playVideo callback:

Ext.getBody().on('click', Preview.Feed.playVideo, null, {delegate: 'a.video'});

When the event is fired we can then replace the contents of the '.item' div with the embed html that we saved in custom data attributes:

playVideo : function(e,t){
  e.preventDefault();
  // Get the parent '.item' div
  var elem = Ext.fly(t).parent('.item');
  // Set the '.items' content to the 'data-embed' value.
  elem.dom.innerHTML = decodeURIComponent(elem.dom.getAttribute('data-embed'));
},

Once a user clicks the thumbnail, the end result looks like this:

While we could add a few other features here, we have chosen to keep it simple. Hopefully you have a good understanding of how all the parts fit together and can build on additional features. Definitely check out the demo and the source code for this project. It's heavily documented and deals with some of the little things like loading notifications.

If you have any questions or comments you can send us a note to support@embed.ly or submit an issue.

Permalink | Leave a comment  »

Last week we switched over to a key based authentication system. It helps us efficiently measure who is hitting the service, and it's simple for developers to incorporate into their apps. Perhaps it is too simple. There is a concern that others can take a key and start using it on their sites. While we do offer oAuth for developers to truly protect their credentials, we want to make it easy for anyone to lock down their key.

We have had the ability to restrict keys by referrer internally for a little while now, but wanted to make sure it was solid before releasing it to the world. Today you will be able to restrict key usage to only the domains that you specify.

Manage Referrers

To get started log into your Embedly dashboard. You will see a new box named 'Manage your Referrers'. This is a list of all the sites that you allow access to.

When you click 'Manage' you will be taken to the Client Referrers page where you can edit and add new referrers. We use a simple wildcard syntax for adding domains much like how oEmbed declares schemas. For example, if you want to allow all traffic from localhost:8000 you would enter localhost:8000*. If you wanted to allow only traffic from subdomains of localhost.com it would look like *.localhost.com*. Take special note of the wildcard at the end. That is required if you want us to match any and all paths for a domain. Otherwise we will only allow traffic from the specified URL.

We also built a simple way for users to test their patterns from within the Client Referrers page. Just input a URL from your site and we will let you know if it matches or not.

We are continuously trying to improve our service based on your feedback. Keep it coming. Next on the list is adding support for matching User Agents. Expect that in the next few weeks.

Permalink | Leave a comment  »

Wouldn't that be a magical page-turner? Now that we have your attention, today we are launching a long-overdue update to our WordPress plugin.

Permalink | Leave a comment  »

It's been three weeks since we mentioned our plans to consolidate api.embed.ly and pro.embed.ly into one API. Now it's time to start talking about our transition plan for that. We would like to make it clear how we will be proceeding and how the changes will affect developers. First things first:

Sign up for an API key here: http://embed.ly/pricing

It's incredibly simple and free! Adding the key to your requests will avoid any service interruption.

On July 20th we will be making the following changes:

• Any developer making requests without an API key will be rate limited to 50 URLs a day. When you reach your limit you will see a 403 forbidden response.
• Requests without a key will be limited to the providers here: http://embed.ly/providers

We are leaving the API slightly open to encourage users to play with it a bit and to avoid breaking a number of apps that currently run on Embedly. By signing up for the Free plan you will get 10,000 URLs a month and access to the oEmbed endpoint for any URL.

We are trying to give existing users enough warning to sign up for a key before we throttle their service back. In order to aid this process we will be giving users a taste of what is to come. On July 6th and again on July 13th we will be turning on those changes we described above for two hours. This will very quickly show existing users how their apps will function after July 20th when this change becomes permanent.

When we launched the reorganization 3 weeks ago we outlined the new feature set of Embedly and slightly changed what endpoints were available at what tier. Users that signed up for Pro prior to the change have been grandfathered in and will have the same features that they originally signed up for.

We look forward to an easy and smooth transition so please reach out if you have any questions or concerns: support@embed.ly

Permalink | Leave a comment  »

oohEmbed is a site that allows you to embed content from 31 different providers through one endpoint. It was developed by Deepak Sarda to facilitate Dumble, his delicious tumblelog. What started as a small open source project grew over the last 3 years and gained a ridiculous amount of traffic. So much so that it went over Google App Engine's free quota limit almost weekly. It's a good problem to have if you are a company, but if you're a single open source developer it's sort of a problem.

In May, Deepak posted a note to the oEmbed mailing list that he was interested in finding a new owner to keep the site going and we jumped at the chance. A month later we are the proud owners of oohembed.com!

This made sense for a lot of reasons. oohEmbed predates Embedly by almost a year and a half and was the basis for what we believed embedding should look like. We have also known Deepak for sometime now and he has been amazing to work with during the transition.

There are a few things that we need to talk about. First off the project is open source and will remain that way. You can find the source on Google code. Embedly will not be supporting the code base going forward, but we encourage developers to.

On July 20th we will be redirecting all traffic from the oohEmbed endpoint to Embedly's API. It's not possible for us to support two codebases and two APIs, so we need to roll oohEmbed into our existing architecture. This is awesome for oohEmbed users because you will be getting the added benefit of Embedly's 218 providers as well as better response times.

We do need to ask a small favor of oohEmbed users. We ask that you grab an API Key and start using it in your requests. It's a 30 second process and will be ready for the switch on July 20th. As an example if your current request to oohEmbed looks like:

http://www.oohembed.com/oohembed?url=http%3A%2F%2Fvimeo.com%2F18150336

All you need to do is add our API key to it like so:

http://www.oohembed.com/oohembed?url=http%3A%2F%2Fvimeo.com%2F18150336&key=aaaaaaa

That's it! You can also switch the endpoint to api.embed.ly/1/oembed to avoid the redirect like so:

http://api.embed.ly/1/oembed?url=http%3A%2F%2Fvimeo.com%2F18150336&key=aaaaaaa

After July 20th oohEmbed users will be held to the same pricing plans and rate limits as every Embedly user. In the meantime we will be footing the bill for oohEmbed so current users will see less downtime.

We look forward to working with the current oohEmbed users and please feel free to reach out to us anytime with questions.

Permalink | Leave a comment  »

While it has been heads down around the Embedly office lately, we are now ready to unveil what we have been working on and what we have accomplished. Today, we are excited to announce two great advances: major changes to the products and a new round of funding!

On any given day more than 1,200 different sites use Embedly with over 7.2 million URLs coming through. Last month, we generated over 2 billion impressions for the embeds we have served. These numbers continue to trend upwards, making us a significant part web infrastructure. Our Pro product is used on sites like TweetDeck, Bit.ly, Hunch, Storify, StockTwits, and Yammer. In addtion, our free API has continued to see adoption all over the web. There is still some confusion regarding Embedly and what API to use; we are about to make that a lot easier.

Today, we are relaunching the service as one API with a unified set of features! Before, the free Embedly API only allowed users to embed content from 218 different providers, while Embedly Pro allowed any URL. These have been merged into one API and we have extended some of the Pro features into the free plan. When you sign up for a key you will be able to embed any URL through Embedly. No RegEx needed!

You can learn more about the feature set of the Embedly API here, but it includes:

• Direct integration with 218 providers
• Rich meta data for any URL
• Analytics
• Mobile Support
• RSS Feeds
• Safe Browse

We believe that making content more accessible to everyone will be a big win for the community as a whole. In order for us to make this transition we are asking that everyone sign up for an API key before July 1st. If you are an existing api.embed.ly user, you can sign up for an API key here.

Before we move on, a few more references:

• The new documentation can be found at embed.ly/docs
• explore.embed.ly has been depreciated for embed.ly/docs/explore
• Existing Pro users can now log into their dashboard at app.embed.ly

Back in July of 2010, we raised a small amount of money to try to make a run at this whole embedding thing. At the time, we were doing 4 million URLs per month. Today, we do that before noon on any given day! We are stoked to announce that we have taken in another 450k to continue growing Embedly from Howard Lindzon, Social Leverage, Venture 51, Adam Schwartz; and a follow-on investment from Betaworks and Chris Sacca. Adam will also be joining our board.

The awesome thing about this round is that two of the new investors are CEOs of companies that use Embedly: Howard at StockTwits and Adam at Articulate. They both have a strong knowledge of the product and know how to turn tech into a business.

We have gone through a lot of transitions over the last year. We started as a hobby product, found our way into an angel round and started generating revenue. We are embarking on by far the hardest part now: turning our small startup into a business. We hope you all will join in on the ride.

Sean

CEO @embedly

Permalink | Leave a comment  »