Embedly Blog

Build or Buy the API

screeley — Fri, 17 May 2013 11:28:00 -0400

Build vs Buy is a classic debate. Do we spend the money on internal resources to build a product or do we just pay someone else to do it for us?

The common approach is to just compare what it will cost internally vs externally. Two Engineers for 6 months vs X dollars for the product.

Rather than thinking about the product itself it’s helpful to think of the intangibles that aren’t inherent in the buy, the things that would be difficult to obtain by building the product yourself.

So what are you really getting when buying?

1. You are buying expertise.

While your two engineers are probably excellent, they haven’t lived and breathed a space for years.

The SendGrid team are experts at email delivery. While most developers can set up their own Email server they can’t tell you exactly why an Email isn’t getting delivered and how you can fix it.

The Zencoder team are experts at quickly converting any video into any format. They know where the browser and mobile market is going and can get you there faster.

The Crocodoc team are experts at converting the many horrors of Microsoft products into HTML5. Every edge case can be properly rendered and look exactly the same on any platform.

2. You are buying time.

Anything that is outside of your core competency should be delegated to someone else. Unless you plan on competing with the company, you should probably just buy their product.

3. You are buying QA.

“Given enough eyeballs, all bugs are shallow”. When one developer finds a bug with an API, the community benefits. It’s almost impossible to catch all edge cases even with great testing practices. With a large community of developers and their users your QA department is almost infinite.

4. You are buying community.

There are a massive amount of talented engineers out there that are already building interesting products using any given API. A quick search of StackOverflow shows that there are 332 questions tagged with Twilio, 208 with SendGrid and 164 with filepicker.io to name a few.

You are in good company when you buy.

5. You are buying scale.

Interesting things start to happen at scale. Embedly’s average response time has dropped dramatically over the past year. As the service has grown and more developers use the product there is a greater probability that a URL requested is already in cache. This allows use to respond quicker and I assume this is also the case with a company like FullContact.

6. You are buying relationships.

There is almost always someone upstream that can cause issues. Twilio has the Phone Companies, SendGrid has Email providers, FullContact has Social APIs, Filepicker has Dropbox and Embedly has Providers. At scale you have probably already cultivated relationships that when something fails you know exactly who to talk to.

Embedly buys MixPanel, SendGrid, FullContact, Zendesk, GitHub and Chargify.

Who are you buying time from?

Improve your drip campaigns with Mixpanel and FullContact

thejohnnest — Thu, 16 May 2013 09:27:00 -0400

By using FullContact to push user profile data into Mixpanel we’re able to make targeting and tracking more powerful. We can target drip marketing campaigns more precisely based on where our users work, as well as their social profiles, location or age.

For those that are unfamiliar, Mixpanel helps you track your users and customers as they use your application, hopefully providing insight into how to increase conversion rates (“revenue analytics”). FullContact is a cool little API that takes an email address and provides lots of information about the person behind that address — his or her social network profiles, places of employment and some basic demographic information.

We are trying to get to a place where we can trigger notifications based on personal attributes instead of just when they signed up. Here is an example of targeting made possible by FullContact and Mixpanel:

The following is a short tutorial on using the FullContact API to push user data into Mixpanel.

First off, this is what a user might look like in your Mixpanel project:

{
    "email": "sean@embed.ly",
    "plan": "basic",
    "date_joined": "May 1, 2012",
    "last_login": "May 10, 2013"
}

This information is probably populated by your frontend via the Mixpanel javascript library, but it’s perfectly reasonable to do it in the backend as well. Fullcontact, however, is a separate API that you’ll have to call on your backend. Looking up an email address is very straightforward via its HTTP API:

$ curl "http://api.fullcontact.com/v2/person.json?email=sean@embed.ly&apiKey=YOURKEY"

{
  "status": 200,
  "likelihood": 0.89,
  "requestId": "aa2f9f2c-57b8-48ab-811b-1be24aa652c8",
  "photos": 
  ...
  "demographics": {
    "locationGeneral": "Boston",
    "gender": "Male"
  },
  ...
}

Take a look at the full response if you’d like to see what kind of information it can provide. Pretty simple, although there are a couple details to keep in mind:

Ocassionally Fullcontact will return HTTP 202 which indicates your search has been queued and that you should retry the request in a few minutes
A status code of 404 means the query has run and no information was found
Fullcontact will rate limit your requests based on your plan

This means you’ll want to do your lookups in batches, which might look something like this:

def fullcontact_lookup(emails, api_key):
    results = {}
    to_lookup = set(emails)
    api_format = 'http://api.fullcontact.com/v2/person.json?email=%s&apiKey=%s'
    while len(to_lookup) > 0:
        # make a copy so we can delete from the set while we iterate
        for email in to_lookup.copy():
            api = api_format % (email, api_key)
            response = json.loads(requests.get(api).text)
            if response['status'] == 200:
                results[email] = response
                to_lookup.remove(email)
            elif response['status'] == 404:
                results[email] = None
                to_lookup.remove(email)
            elif response['status'] == 202:
                # keep it around to give it another go
                pass
            else:
                raise ValueError('Unexpected response from fullcontact: %s' %
                    json.dumps(response))
        # give fullcontact a minute to run some queries
        time.sleep(60)
    return results

Now we need to actually stuff this data into Mixpanel. Unfortunately, naively running fullcontact_lookup against your entire database every night will be expensive, as Fullcontact charges per successful request. What we can do is add a property to each person called, say, fullcontact_version that keeps track of who has been queried. This can be an integer that gets incremented each time you change your schema, so you can just run the people with fullcontact_version != current_version. Treat the code below as pseudo-code; if you’d like to see the nitty gritty Mixpanel API details check out their docs.

def go(fullcontact_version):
    to_update = get_with_different_version(fullcontact_version)
    all_info = fullcontact_lookup(to_update)
    for person in to_update:
        info = all_info[person['email']]
        person['fullcontact_version'] = fullcontact_version
        if info:
            person['age'] = info.get('demographics', {}).get('age')
            # add more properties as you see fit
        save_person_to_mixpanel(person)

And there it is! With this small amount of effort you can get greater insight into who your users are and the ability to communicate with them more efficiently.

Display: An Image Proxy and Resizing API for Web Images

artgibby — Wed, 15 May 2013 10:36:00 -0400

Our newest product is an Image Proxy and Image Resize API, fondly referred to as Display. This service is being thrown into the lime-light today to highlight its simplicity and usefulness for providing optimized 3rd party images to your web or mobile application.

Display is useful in many situations, but we find that some of the most important features include its use as a proxy to provide SSL images and its ability to resize images, which not only decreases the file size, but also allows you to scale them to fit and load quickly in any mobile or web browser.

The Image Resize API is a great complement to any 3rd party media API aggregating photos and images of various sizes. It integrates directly with our own Embed and Extract APIs to provide SSL images and resizing automatically.

See the Display image resize feature in action in the Resize demo or try some examples with Embedly JQuery.

Images make your site more engaging, but they slow down page load time, and it’s hard to optimize them for different screens. Display solves that problem. Sign up to get started with Display.

Vine's not a rocket ship, but it's doing well outside of Twitter

screeley — Fri, 10 May 2013 10:30:00 -0400

Vine is doing well outside of Twitter, better than it’s competitors, but it’s not a rocket ship.

We set out to measure the rise of Vine as a provider and see how it compares to other mobile video sharing tools. Here are the total URLs that we have seen posted to Embedly’s API since Vine launched on January 24, 2013:

You’ll see a nice bump post launch and the classic exponential growth that you would like to see, but that wore off after about a week. Vine’s competitors are barely a blip on the radar, so let’s look at this graph on a logarithmic scale:

Vine still dominates, SocialCam seems to do a bit better than Qik and Viddy who are about equal.

How many URLs we get through the system is not that important because that can change on any day due to one client hammering us with all Vine URLs. It’s more about many different developers are requesting a certain provider along with velocity.

The following chart scores providers based on the number of URLs and the diversity of developers. It is a little more telling of how well Vine is doing:

Vine is trending up, while the rest have a slight trend downwards.

Vine is getting some good distribution throughout the long tail and it’s pretty clear why Viddy is giving back 18 million dollars.

Finally, to put this in perspective we did 1,345,449 YouTube URLS, 243,897 Instagram URLs and 25,892 Vine URLs yesterday. Vine still has a long way to go.

Ember at Embedly Tutorial: Models and Ember Data

dokipen — Thu, 09 May 2013 10:11:00 -0400

In this fourth post in our series on Ember, we will continue to work from the development environment that we worked on in the first three posts.

This installment will focus on models, backed by RESTful storage, using ember-data. First, a giant, red, warning: ember-data is not production ready. It is under active development and changes often. There are bugs and limitations and you’ll have to wait for them to be fixed. We at Embedly, in our youthful exuberance, decided to ignore these problems and blaze forward into the future! If you do decide to follow us, stay up-to-date with breaking changes at BREAKING_CHANGES.md.

As usual, you can find the complete code for the demo project at github. We won’t cover every code change here, just the ones directly related to ember models and ember-data.

If you remember, in our last installment, we created a team page. We used fixtures to provide data to the template. Let’s switch out these fixtures for some data provided by a RESTful api. We’ve built a simple api server in node.js to connect to. It is only an example and does the bare minimum to get us up and running. Please don’t consider using it for anything remotely serious.

Adding ember-data

We are using bower to manage our javascript dependencies. There is an ember-data wrapper, called ember-data-shim, made specifically for dependency management. If you use it, make sure it is relatively up-to-date when you add it to your own project. Here is our new component.json.

    // bower.json

    {
      "name": "demo",
      "version": "0.0.0",
      "dependencies": {
        "modernizr": "~2.6.2",
        "jquery": "~1.9.1",
        "handlebars": "1.0.0-rc.3",
        "ember": "1.0.0-rc.3",
        "ember-data-shim": "0.0.12"
      },
      "devDependencies": {}
    }

Running bower install will install the dependency into our app/components path. Now we just have to add ember-data.js to our app/index.html.

    // app/index.html

    - snip -
    <script src="components/ember/ember.js"></script>
    <script src="components/ember-data-shim/ember-data.js"></script>
    - snip -

If everything went as planned, we should be able to start the app, open it in the browser, and check to see if there is a global DS object available in the browser’s javascript console.

Configuring ember-data

There is a little bit of configuration to do to get ember-data working. We’ll add it to app/scripts/main.js. The first part tells ember-data what version we are planning on using. This is important since the API has breaking changes between revisions. We don’t want to accidentally upgrade before we are ready. The second section tells the RESTAdapter the base URL for our RESTful api.

    // app/scripts/main.js

    - snip -
    App.Store = DS.Store.extend({
      revision: 12
    });

    DS.RESTAdapter.reopen({
      url: '/rest/1'
    });

    - snip -

You may be asking yourself, what is the RESTAdapter, and what other adapters are there? Ember-data comes packaged with a RESTAdapter and FixtureAdapter. The FixtureAdapter provides an in memory browser store and is mostly useful for testing and development.

FixtureAdapter was an option when writing this post, but I also wanted to show the expected responses and results of the API calls, so that when you go to implement your own RESTful service, you’ll know what to expect.

Defining Our Model

Defining our model is very simple. First we’ll remove all our “Fixtures” from app/scripts/modules/about.js, and replace them with our model definitions in app/scripts/modules/models.js.

    // app/scripts/modules/models.js

    App.Org = DS.Model.extend({
      name: DS.attr('string'),
      description: DS.attr('string'),
      members: DS.hasMany('App.Member')
    });

    App.Member = DS.Model.extend({
      name: DS.attr('string'),
      org: DS.belongsTo('App.Org')
    });

    // app/index.html

    - snip -
    <script src="scripts/modules/models.js"></script>
    - snip -

Here we are only using the string attr type, but string, number, boolean, and date are also available. It is also possible to define your own attr types as well, but we won’t cover that here.

We are also defining a many-to-one relationship between Member and Org. We really only have one Org, but who knows how our site will grow in the future. It is possible to declare many-to-many relationships, but let’s keep things simple.

Wiring up our Route and Controller

Let’s update the about page to use our new model, replacing our old fixture code.

    // app/scripts/main.js

    App = Em.Application.create({
      rootElement: $('#app'),
      orgId: 0
    });
    - snip -

    // app/scripts/modules/about.js

    - snip -
    App.AboutRoute = Em.Route.extend({
      model: function() {
        return App.Org.find(App.get('orgId'));
      }
    });

    // app/templates/about.handlebars

    - snip -
    {{render "team" controller.members}}

We’ve configured our orgId in the App object. This is a good place to put site-wide configuration. Next we replaced our fixture with a call to App.Org.find() in our Route#model. ember-data will return an empty object, then switch in the Org when it’s ready. Since Ember.js is so wonderful, it will re-render the relevant parts of the view on update. Finally, we replace the members fixture in the about.handlebars with the members field of the Org object, which is an Array of organization members.

Now our application behaves exactly the same as it did before we implemented ember-data, except that it is backed by a RESTful API. Pretty simple, eh? But not very interesting, we’ll have to add some CRUD operations to truly demonstrate ember-data’s power. Let’s take a look at, what’s happening between the server and client before we move on to mutating the data.

Retrieving records using ember-data is extremely simple. One thing to keep in mind is that it’s asynchronous. Usually you can safely ignore that fact, due to the way ember’s event system works, but sometimes it can bite you, so keep it in mind. We retrieve our Org with the line App.Org.find(App.get('orgId')). Let’s take a look at what our server returns.

  // Request

  GET /rest/1/orgs/0

  // Response

  {
    "org": {
      "name": "jubarian.org",
      "description": "A non-profit organization of people...",
      "member_ids": [ 0, 1, 2, 3, 4, 5, 6 ]
    },
    "members": [{
      "org_id": 0,
      "name": "Nina",
      "id": 0
    }, {
      "org_id": 0,
      "name": "Kawan",
      "id": 1
  - snip -
    }]
  }

There are a few things to be aware of here. First, we can see that our API puts the payload of the org object into a field called org (singular). This is where ember-data will look for the Org data. If we had made a more general query that returned a list, it would be orgs in the plural form. Second, we are also returning all the members data with the response. This is referred to as side-loading, and it is optional. In our case, since we have the luxury of a very small application with well defined requirements, we know that this is the most efficient way to do things. If we didn’t side-load, ember-data will make another call to retrieve the data when it’s needed.

Notice that the members field is plural, it’s a list, and ids are included in the objects. Also notice the serialization of the data fields. Ember expects all serialized field names to be underscore separated, even though it expects camelcase in the javascript models. It also expects a pluralized member_ids field for the one-to-many relationship, and a singular org_id for the many-to-one relationship.

All of these behaviors can be modified. Ember-data is very customizable, but unless you are dealing with a legacy API, it’s always best to conform with the frameworks expectations.

Ember Data’s Protocol

Let’s give a brief overview of what ember-data expects to send and receive during different transactions. Understanding this protocol will not only help you implement your own persistent backend, but also is invaluable for debugging ember-data errors, which are often the result of some unexpected data. Many of these details can be modified by configuring the RESTAdapter, but again, if possible, it’s better to stick with the standard. Keep in mind that if you have an existing API that is much different, it might be worth writing your own adapter. We won’t cover custom adapters in this post.

GET one

Call

find(:id)

HTTP Method

GET

URL

/<plural model>/:id

Example Request

GET /orgs/0

Example Response

{
  "org": {
    "name": "jubarian.org",
    "description": "A non-profit organization of people...",
    "member_ids": [ 0, 1, 2, 3, 4, 5, 6 ]
  }
}

GET many

Call

find({:attr: :value}

HTTP Method

GET

URL

/<plural model>?:attr=:value

Example Request

GET /orgs?name=jubarian.org

Example Response

{
  "orgs": [
    {
      "id": 0,
      "name": "jubarian.org",
      "description": "A non-profit organization of people...",
      "member_ids": [ 0, 1, 2, 3, 4, 5, 6 ]
    }
  ]
}

With mutations, we need to commit our changes before they will be sent to the server, either through the records transaction or the store. If we change our minds before we commit, we can invoke rollback instead of commit.

POST

Call

var record = Model.createRecord({...});
record.get('store').commit();

HTTP Method

POST

URL

/<plural model>

Example Request

POST /members

Example Body

{"member":{"name":"Billy","org_id":0}}

Example Response

{"member":{"name":"Billy","org_id":0, "id": 7}}

PUT

Call

var record = Model.find(:id);
record.set(:name, :value);
record.get('store').commit();

HTTP Method

PUT

URL

/<plural model>/:id

Example Request

PUT /members/7

Example Body

{"member":{"name":"William","org_id":0}}

Example Response

{"member":{"name":"William","org_id":0}}

One thing to note is that DELETE MUST return valid json, or you will get an error. I consider this a bug, so hopefully it changes in the future.

DELETE

Call

var record = Model.find(:id);
  
record.deleteRecord();
record.get('store').commit();

HTTP Method

DELETE

URL

/<plural model>/:id

Example Request

DELETE /members/7

Example Response

{}

Creating a Team Admin Page

Next, let’s put it all together and build a member management section for our site. First let’s setup the routes. We’ll have three pages; an index, a new member form and an edit member form.

    // app/scripts/main.js

    - snip -
    App.Router.map(function(){
      this.route('about');
      this.resource('members', function() {
        this.route('new');
        this.route('edit', {path: '/edit/:id'});
      });
    });
    - snip -

Now, let’s build the Routes and Controllers.

    // app/scripts/modules/members.js

    App.MembersIndexRoute = Em.Route.extend({
      setupController: function(controller, model) {
        App.Org.find(App.get('orgId')).then(function(org) {
          controller.set('content', org.get('members'))
        });
        controller.set('content', Em.A());
      }
    })

    App.MembersEditRoute = Em.Route.extend({
      serialize: function(obj) {
        return {id: obj.get('id')};
      },
      model: function(params) {
        return App.Member.find(params.id);
      }
    })

    App.MembersNewRoute = Em.Route.extend({
      model: function(params) {
        return Em.Object.create({org: App.Org.find(App.get('orgId'))});
      }
    })

    /*
     * Wrap member model objects in simple interface callable
     * by the view.
     */
    App.MemberController = Em.ObjectController.extend({
      "delete": function() {
        this.get('content').deleteRecord();
        this.get('store').commit();
      },
      edit: function() {
        this.transitionToRoute("members.edit", this);
      },
      save: function() {
        this.get('store').commit();
        this.transitionToRoute("members.index");
      },
      cancel: function() {
        this.get('store.defaultTransaction').rollback();
        this.transitionToRoute("members.index");
      },
      create: function() {
        this.set('content', App.Member.createRecord(this.get('content')));
        this.save();
      },
      cancelNew: function() {
        this.transitionToRoute("members.index");
      }
    })

    App.MembersIndexController = Em.ArrayController.extend({
      itemController: "member"
    })

    App.MembersNewController = App.MemberController.extend({});
    App.MembersEditController = App.MemberController.extend({});

    // app/index.html

    - snip -
    <script src="scripts/modules/members.js"></script>
    - snip -

If you’ll notice, in MembersIndexController we forgo using the model callback in favor of the setupController callback. This is because we are passing in a field of the result of an asynchronous operation. MembersIndexController needs an Array right away to set itself up. If we called something like Members.find({org_id: 0}) then ember-data would have returned an empty array right away and we would have been fine, but I wanted to showcase a situation where the asynchronous nature of ember-data can leak through, so it’s important to be aware of. Once the call has returned, we can safely swap the result into the controllers content field and everything will be updated as expected.

We’ve also introduced a new controller callback serialize, in the MembersEditController. If your Route has parameters, it is important to provide a way to map the object it represents to the URL parameters. This way when we transition to the route and pass an object directly to it, ember can form a linkable URL out of it. Our serialize method maps the id attribute of the model to the id parameter in the edit URL.

Our MembersIndexController introduces the concept of itemController. itemController wraps each item in an ArrayController, making it easier to perform actions on individual items from the view. In our case, it will allow us to easily delete and edit the individual items.

We used inheritance to get all the functionality of MemberController into MembersEditController and MembersNewController. Since we haven’t added any view logic, we are using the default generated views. Let’s finish up by adding templates for our member management pages, and a link from the about page.

    // app/templates/about.handlebars

    - snip -
    <p>
      <a href="#" {{action toggleTeam}}>{{toggleTeamLabel}}</a>
      {{#linkTo members.index}}manage team{{/linkTo}}
    </p>
    - snip -

    // app/templates/members.handlebars

    <p>{{#linkTo "about"}}about{{/linkTo}}</p>
    <p>{{outlet}}</p>

    // app/templates/members/index.handlebars

    <p>List Members</p>
    <ul>
    {{#each controller}}
      <li>
        {{name}} <a href="#" {{action delete}}>delete</a> <a href="#" {{action edit}}>edit</a>
      </li>
    {{/each}}
    </ul>
    {{#linkTo "members.new"}}add member{{/linkTo}}

    // app/templates/members/edit.handlebars

    <p>Edit Member</p>
    <p><label>
      Name
      {{view Em.TextField valueBinding="name"}}
    </label></p>
    <p>
    <a href="#" {{action save}}>save</a>
    <a href="#" {{action cancel}}>cancel</a>
    </p>

    // app/templates/members/new.handlebars

    <p>Create Member</p>
    <p><label>
      Name
      {{view Em.TextField valueBinding="name"}}
    </label></p>
    <p>
    <a href="#" {{action create}}>save</a>
    <a href="#" {{action cancelNew}}>cancel</a>
    </p>

Security

Obviously, if this were a real application we wouldn’t want just anybody to be able to modify our member list. We’d have to implement some security, but that is outside the scope of this series of articles. One thing to be aware of is that your RESTful API is publicly available, so make sure your security is implemented there, and not on the frontend. Your ember app should be aware of security, but your API should enforce it.

Error Handling

Error handling in ember-data leaves much to be desired, and this is where the fact that it’s a beta project is really obvious. It’s possible to build some error handling via callbacks and states on model objects, but it’s difficult and ever changing. Even finding documentation on the relevant callbacks and states is difficult. It usually involves reading the ember-data code base. We’ve chosen to skip it in this article. It would be of little lasting value, since we expect it to mature rather quickly.

Conclusion

I hope this article, and this series has been able to give you a good high level understanding of ember and ember-data and get you quickly started on the right path. Even if you choose not to use ember-data and roll-your-own, I hope this article has given you some ideas on how to build it. This wraps up our series on Ember at Embedly. Now go create something awesome!

Bulk Usage Tiers

artgibby — Wed, 01 May 2013 14:00:00 -0400

Today we are unveiling bulk usage tiers across all products. These tiers will provide you with a discounted cost as your use of the service increases—revolutionary! These changes will be available immediately and automatically occur for anyone using one of our new products.

If you’re getting the news late, we announced Products back in late March as a way to create the best front-end tools and services on the web. One of our main goals with the products was to scale, and pricing seemed like a good place to start.

You can check out the usage tiers and use our handy calculator on each product’s pricing page—Display, Embed, and Extract. Here’s a glimpse of the Embed product’s usage tiers:

Have any questions? Just give us a jingle: sales@embed.ly. If you haven’t tried any of our products, sign up for our free intro.

User Spotlight: Summary.io

ninerr — Mon, 29 Apr 2013 16:38:00 -0400

This week we’re spotlighting recently launched summary.io, and the founder, Sam Sneeling.

Here we go, what is the mission behind summary.io?

Summary.io was built as a proof of a concept that you can summarize a large amount of news in a short period of time. The overall goal for our first version is to enable people to find articles that they want to read. The technology that powers it can really be applied to almost any body of text. We are looking at how summarizing text can make an impact in everyday lives.

How does Embedly help with that mission?

Embedly really helps us look good (by providing images for our news summaries). As of right now, Embedly is used as a fallback, but it works consistently. Really helps overall to have a reliable backend.

What would you like to see next from Embedly?

I think Embedly has a lot to offer. Digging into your docs, your kitchen sink api is really quite incredible. I look forward to seeing that api become more available for a lower cost so that small developers can create really cool stuff.

If you’re interested in using our natural language processing and text extraction tools, check out our new product, Extract.

Startup Institute Boston Expose

ninerr — Fri, 26 Apr 2013 18:27:05 -0400

Yesterday, Kawan and I ventured to the Harvard Innovation Lab on a beautiful day in Boston to see the graduating class of Startup Institute Boston’s Spring Track. 50 hard-working and passionate individuals gave unique, expressive one-minute pitches in front of a crowd full of Boston based companies looking to hire.

After each pitch from web developers, sales, marketing, and product design students, a networking session commenced. We met a ton of talented folks and had a great time doing it. As each student talked about their accomplishments, Kawan and I would sort of look at each other with wide eyes because we were so impressed by the amount of life experiences and ambition they had. We’re excited to be connecting with the Boston startup community.

If you had a fun time and want to chat with us some more, or didn’t get to meet myself or Kawan, venture out to sunny North Station next Friday from 11am to 1pm for our stop on the Startup Walkabout. We’ll be having lunch and beers, so come out and meet the rest of the team!

Separating from the Flock

screeley — Thu, 25 Apr 2013 13:33:00 -0400

As of May 24th we will no longer support Twitter Embeds.

Embedly is in violation of section 1.4.A of the Twitter Terms of Service by resyndicating their oEmbed API response. With the release of v1.1 of their API we lost our white list status and have been using a stop gap ever since.

Yesterday we got the official word that we must fall in line after a month of trying to work something out.

We had a good run and a pretty good relationship with Twitter.

We were white listed to make a high volume of API calls.
We were listed as a method to embed tweets on dev.twitter.com before their official embed was released.

Embedly supports embedding content from over 250 different sites and this is the first time that we have ever had to remove a provider. Every other provider we have talked to views Embedly as a method of furthering their reach. More sites where there content can be shared and grow.

So, what does this mean for Embedly users?

Twitter’s Terms of Service does allow us to pass back the tweet id, but we have chosen not to include this in our oEmbed response. The tweet id is fundamentally useless for embedding tweets, especially when you already have the URL.

If you would like to hit Twitter’s oEmbed API directly, please refer to their documentation. It requires oAuth authentication, so disregard the o in oEmbed.

On May 24th we will remove Twitter from our providers list, so if you use the Embedly’s Wordpress Plugin you will need to update your providers list then. Other than that, things will just fail naturally.

I apologize for this one and wish we could have come to a better resolution with Twitter.

The good news here is that Twitter represents only about 0.4% of URLs Embedly processes on a given day.

If you have any questions or concerns you can contact me directly at sean@embed.ly.

Sean

Boston Startup Walkabout with Embedly

ninerr — Wed, 24 Apr 2013 12:48:00 -0400

Next Friday, May 3, we are opening our doors to the StartupWalkabout community for a meet and greet with our team from 11am to 1pm. Serving up lunch and some of our personal favorite brews, we’re making our presence known in Boston.

The Boston Startup Walkabout is a 4-day long city-wide event for job seekers and members of the tech community. Over the course of the week, some of Boston’s best startups will be opening their doors to network, enjoy some treats and show why you should join their team. With some great names such as TechStars, MassChallenge, and PayPal, you will be wowed by some of the opportunities these Boston startups have to offer.

If you’re in the area and want to get to know the local startup community, looking for a job, or just want to see what we’ve been up to, swing by and grab a bite. We want to get to know our neighbors too so look out for an Embedly employee at some of the walkabout events! Take a look at our jobs page to get a better understanding of what we’re looking for.

Final Note: Space is limited to 20, so snag your spot quick.

User Spotlight: Blain Smith of Adjacent Concepts

ninerr — Mon, 22 Apr 2013 16:12:00 -0400

A few days ago I came across a tweet:

Quickly embed content into @statamic with this @embedly plugin github.com/blainsmith/Sta…

— Blain Smith (@blainsmith)

April 15, 2013

I was pretty pumped because I enjoy seeing what people can do with our API and this was a perfect example. So, I reached out to Blain with a few questions…

What sorts of projects do you generally work on?

My projects are all over. I’ve done a lot of small business websites, SaaS web apps, custom web apps, and large scale websites. Recently I joined MadGlory Interactive as a partner where we build web communities for the gaming industry. Everything from franchise websites to forums and custom leader boards of stats.

What inspired you to create the plugin for Statamic?

Aside from being close friends with the creators of Statamic, I really believe in it. I also believe that in order for any new CMS to be successful it needs a really broad range of open source plugins available to extend its functionality. Embedly is a great service to extend Statamic to allow its user to easily publish media from all over the web without being technical. Embedly deserves to be a part of every CMS to make embedding easy.

What did you like about using the API?

Not only is the API easy to use, the extensive list of media providers is amazing. It really makes integrating media from other sources such a seamless and uniform experience.

What would you like to see from Embedly after working on this project or in general?

In general, I would love to see Embedly in the media more. I read a lot of articles on Hacker News, Smashing Magazine, and Slashdot, and other sources, but I haven’t seen much from Embedly on there. Embedly really deserves a lot of recognition for the service it provides.

Wow, what encouraging words! Make sure to take a look at the plugin.

This post is part of our spotlight series. Let us know what you’ve created with Embedly- small project or a massive endeavor- we want to hear about it. If you haven’t tried the API yet, you can get started now.

Encryption Project Uses Embedly Extract to Preprocess Articles

whichlight — Wed, 17 Apr 2013 15:16:00 -0400

Embedly was selected to present our APIs and offer support as an ambassador at the 2013 hackNY. There was lots of energy in the room, with a focus on projects being awesome, rather than chasing hockey stick growth. They even had a hardware hacking room with 3D printers.

One of the projects, Edison Encryption, used the Embedly Extract API as one of the steps for an encryption algorithm. The team consisted of Eric Schles, Dan Cohen, Chris Williams, and Danny Padawer.

Details of the algorithm are provided on Github.

The algorithm currently adds padding to plaintext in such a way that makes cryptanalysis more difficult. This is because it pads based on a given corpus (chosen by the encrypter) and mimics that frequency. This makes discerning the true plaintext difficult. In order decrypt, you need a series of offsets, for each actual plaintext character. By combining both of these, we have an easy to decrypt and relatively cheap encryption algorithm.

When I saw the project presentation I was curious how Embedly was used in it. Eric got back to us, mentioning that Embedly Extract was used specifically to parse an article and get the content. From Eric:

We used embedly to get corpus’ of a specific kind. We then did frequency analysis(similar to what is done in cryptanalysis) on the corpus in order to determine likely frequency patterns, making it difficult to tell which letters are part of the plaintext and which are part of the ciphertext.

According to Eric, the team doesn’t plan on stopping now, and has some pretty admirable ambitions.

We plan to expand this hack and turn it into a cryptographic standard that makes the internet a safer better place.

When we built Extract, we knew it would be used in natural language processing applications, but we never imagined it would be used for encryption. Pretty cool seeing how people use the Extract API!

If you’d like to start using Extract, you can sign up free, and get started with some demos and tutorials.

HackNY was a great time. If you’re having a hackathon, let us know, we’d love to come by.

Update (04/18/13): Danny got back to us on what it was like using the Extract API:

We only used the extraction API, but it was very nice, easy, simple to use. It returned everything we needed.

No complaints, and it worked quickly, which is always nice at a Hackathon - minimal time was needed to get it doing everything we needed.

Great to hear!

Ember at Embedly: Views and Controllers

dokipen — Mon, 15 Apr 2013 09:53:00 -0400

The third post in our series on Ember will continue to work from the development environment that we worked on in the first and second posts.

As promised last time, this post will dive into views and controllers. Here is where the magic happens. We’ll also touch on models, but we’ll save the bulk of the models work for our next post.

Controllers

This time let’s try and fill in some details on the about page. Since it needs to be about something, let’s create a fictitious organization called Jubarian. We’ll create a simple controller to display information about the organization and add it to index.html.

    // app/scripts/modules/about.js
    App.AboutController = Em.Controller.extend({
      name: 'jubarian.org',
      description: 'A non-profit organization of peopled dedicating to spreading joy.'
    });

    // app/index.html
    __snip__
    <script src="scripts/modules/about.js"></script>
    __snip__

    // app/templates/about.handlebars
    <h1>About {{name}}</h1>
    <p>{{description}}</p>

This works, but it’s not very MVC. We shouldn’t be adding our data directly to our controller. Our controller should only contain data directly relevant to display concerns.

Controllers in Ember are a little different then what you may be used to in more traditional MVC frameworks. Usually, they are implemented as proxies to a model and passed to the view using an ObjectController or ArrayController. The view will never interact with the model directly, only through the controller proxy. This allows us to have a clean separation of data concerns and display concerns. The controller decorates the model with display logic. Let’s implement something to make things more clear.

ObjectController

There are two main types of controllers that are most common in Ember applications, the ObjectController and the ArrayController. In this case, we’ll use an ObjectController, since there is only a single organization being displayed. We’ll create a naively simple model to hold our data and connect it to the controller in the router.

     // app/scripts/modules/about.js
     App.Fixtures = App.Fixtures || {};

     // our model
     App.Fixtures.Org = {
       name: 'jubarian.org',
       description: 'A non-profit organization of peopled dedicating to spreading joy.'
     }

     App.AboutController = Em.ObjectController.extend({})

     App.AboutRoute = Em.Route.extend({
       model: function() {
         return App.Fixtures.Org;
       }
     });

Properties of Ember objects aren’t set directly. We always use set(name) and get(name) functions to allow Ember to notify the objects observers of property changes. This is what allows Ember to automatically update views when properties change. Since our Org object is not an Ember object, Ember won’t be able to detect changes to it. We’ll cover a more robust way of doing thing in the next post, when we cover models and ember-data.

We’ll need to wire up our Org to the AboutController. We do this by defining a method function property on our AboutRoute object. ObjectControllers are different then a regular controller because they proxy all of their undefined properties to the object defined as their content property. Ember will automatically call the model function and set the result to the content property of the controller.

Our controller, as it stands, doesn’t do anything. Before we created it we were using an auto-generated controller of type Controller. Since it wasn’t an ObjectController, it didn’t proxy to the content property. Now that we’ve defined it as an ObjectController, all undefined properties pass through to the content property, our org fixture.

Let’s add some state to the display so the controller has something to do. We’ll add a list of team members with a toggle for displaying or hiding it. This will also give us a chance to setup an ArrayController and see how that works. But first let’s just create a static view. And modify the AboutController to handle the team display logic.

      // app/templates/team.handlebars
      <ul>
        <li>Bob</li>
        <li>Andy</li>
      </ul>

      // app/templates/about.handlebars
      _snip_
      <p>
        <a href="#" {{action toggleTeam}}>{{toggleTeamLabel}}</a>
      </p>

      {{view App.TeamView isVisibleBinding="teamVisible"}}


      // app/scripts/modules/about.js
      _snip_
      App.AboutController = Em.ObjectController.extend({
        teamVisible: false,
        toggleTeam: function() {
          this.set('teamVisible', !this.get('teamVisible'));
        },
        toggleTeamLabel: function() {
          var vis = this.get('teamVisible');
          return vis ? 'hide team' : 'show team';
        }.property('teamVisible')
      });

      App.TeamView = Em.View.extend({
        templateName: 'team'
      })

You’ll notice that we had to explicitly create the TeamView object. That is because it wasn’t generated through the router so the template name isn’t automatically discovered.

Properties

toggleTeamLabel is a good example of a dynamic property. When controller.get('toggleTeamLabel') is called, the function will be used to generate toggleTeamLabel’s value. To denote this, the property function is called on the function. The property function takes a list of arguments that define it’s dependencies. This is important for observers of the property, and to allow Ember to cache the property result. Ember knows that when teamVisible changes, so does toggleTeamLabel.

In our about template, we have created an anchor tag that fires toggleTeam on AboutController whenever it is clicked using the action helper. We are using the toggleTeamLabel as the display name of the tag. We use the view helper to insert the TeamView into our template. isVisibleBinding="teamVisible" is a special attribute that binds the isVisible attribute of TeamView to the teamVisible property of the current scope, which happens to be AboutController. Now, whenever teamVisible changes, so will isVisible. isVisible is a builtin property of every View that controls whether or not it is visible.

ArrayController

Now let’s convert the static template, team, into a proper ArrayController. Since we want the view to have it’s own controller, and not share a controller with AboutView, we’ll change it from a {{view}} to a {{render}}. This will allow us to wire up the array of team members to an ArrayController and associate it with the view.

    // app/modules/about.js
    App.Fixtures.Team = [
      { name: 'Bob' },
      { name: 'Andy' },
      { name: 'John' },
      { name: 'Kawan' },
      { name: 'Nina' },
      { name: 'Sean' },
      { name: 'Art' },
    ];
    
    _snip_

    App.Team Controller = Em.ArrayController.extend({
      needs: 'about',
      about Binding: 'controllers.about'
    });

    App.TeamView = Em.View.extend({
      isVisibleBinding: 'controller.about.teamVisible'
    });

    // app/template/about.handlebars
    _snip_

    {{render "team" App.Fixtures.Team}}

    _snip_

    // app/templates/team.handlebars
    <ul>
      {{#each controller}}
        <li>{{name}}</li>
      {{/each}}
    </ul>

The render helper will automatically find the correct view and controller based on the first parameter, the template name. The second parameter will be used as the model.

The way we are binding AboutController’s teamVisible property to TeamView is a bit contrived, but it shows how controllers can be bound together. We declare that TeamController needs AboutController and that the about property of TeamController is bound the AboutController. We then bind TeamView’s isVisible property to AboutController’s teamVisible property.

Since TeamController is now an ArrayController, we can iterate over it’s content with the each helper. each will iterate over the collection putting it in scope for the contained template, in this case <li>{{name}}</li>.

Conclusion

With a few lines of code we’ve made a dynamic, data driven about page and demonstrated the power of Ember views and controllers. You can find the source for the project on github This wraps up our brief intro to views and controllers. Our next blog in the series will give an introduction to ember-data, a web ORM for ember. See you next time!

User Spotlight: FetchNotes

ninerr — Wed, 10 Apr 2013 12:02:00 -0400

Our users are what keeps us going so we’ve decided to start a weekly user spotlight series to showcase your projects and creations.

First up is Fetchnotes. An app that’s core mission is to make productivity as simple as a tweet. When using Fetchnotes, your tasks are grouped by hashtags, as in #todo, #email, #groceries, and #buy. As you place hashtags on each task, they are grouped by the hashtag and you can access them more clearly via your phone or dashboard. Pretty slick.

So, how do they use Embedly? Fetchnotes founder and CEO, Alex Schiff filled us in,

“Embedly makes it easy for our users to act on the notes they take, so it directly supports that goal of helping people execute. When they save a YouTube link, Embedly allows them to watch it from directly within our app - no extra steps. When they save an article to read later, it previews it and gives them relevant context. It cuts down on the steps from “I want to…” to “done.”

The original feature of Embed being put to use, to help you finish that growing to do list that’s probably making you sick.

Last thing we’d like to add is what Fetchnotes is hoping to see from us in the near future. We build for you, so when you ask us to create something we always try our best to deliver. What is Fetchnotes hoping for?

“One thing I’d love to see from Embedly is to pull in similar results without needing the link itself. So rather than needing the link to a song on Rdio to pull in the preview, it would be awesome if it could find it based off of having the artist name or song in the text.”

We’ll be sure to add that to our to do list (cleverly placed statements are my speciality, look out). Awesome to see such a dynamic app using our API and we hope to see more. Make sure to take a look at Fetchnotes to kickstart your productivity. And if you’re using Embedly, let me know, I’ll be diligently writing posts for as long as you contribute! Feel free to email me directly at nina@embed.ly.

Interested in what you could create with Embed? Check it out here, mosey on over to the sign up and start using Embedly for free.

Embedly at the ML/NLP Hackathon w/ Traackr and Hack/Reduce

ninerr — Mon, 08 Apr 2013 13:38:00 -0400

As you may know, we co-sponsored a hackday with Traackr and Elasticsearch at Hack/Reduce. A one-day event centered around Machine Learning and Natural Language Processing that turned out to be a hit.

The event host, George Stathis of Traackr had a few favorites but one stood out to us: Alex Lambert from Spindle. He processed articles from the Economist from the past 5 years and rated them on 3 different mood scales from the Affective Norms for English Words (ANEW) word scores to see if indeed the world is coming to an end, no joke. Thankfully, he came to the conclusion that the world is not ending, so we are all safe…for now.

Embedly’s own Andy and Art created their own “NFL coach”. This one hit close to home as many of the guys in the office, including my lady self, are pretty big NFL fans. They took 10 years worth of very detailed NFL play data and used it to train an ML algorithm to predict what play should be called. Once it was trained, they tested to see how often it made the same call as a professional coach would. The results? Out of 44 thousand plays a year in the NFL, their algorithm would give 1 out of 11 options, out of all of those options, they would get around 22 thousand right. So 50% for a bot acting like a pro football coach? Not bad.

John and Kawan gave a talk at the beginning to get the ideas going. They went in depth on how we use machine learning and NLP in-house to build some of Embedly’s features, including parts of our new product, Extract.

The winner of the Best Use of the Embedly API was Sands Fish. I had the pleasure of interviewing him about his project.

Sand’s idea was to use a large amount of data that he had in the form of 15,000+ tweets from this year’s SXSW conference and “write a service to aggregate all of the tweets containing #SXSWi and #SXSWedu”. He said, “my main goal was to enrich the data as much as possible so I could paint a more complete picture of what happened [at the conference] and get some insight into what ideas, people, and companies were being talked about the most”. Pretty nifty.

“The embedly API was a great fit…while I was only able to enrich tweets that contained links, there were more than 5,000 in this category, and for each of them, the API provided me with a wealth of data about the resources linked to in the tweet”. Sands created an image-wall with the data, keeping time in mind, this was the most he felt comfortable creating. Sands mentioned that he had “many ambitions for the data-set” and you better believe we’re excited to see what he comes up with given the luxury of time, post hackday.

Verelo's Public Shaming

screeley — Mon, 08 Apr 2013 10:16:00 -0400

As reported by Pingdom, Embedly’s uptime for the month of April has been 100%. We pay Pingdom to manage these checks and have configured them to properly monitor the API’s health.

We were pretty surprised when Verelo started reporting on Twitter that we were down. If you have never heard of Verelo (neither had I), they monitor uptime like Pingdom and were recently acquired by Dyn.

Apparently they are also into public shaming to try to drum up business, so we are going to return the favor.

Verelo runs a site called whatsdowntoday.com where they hook up their services to a bunch of sites without notifying them first. They ping them and tweet out at @verelostatus when one fails to pass it’s status check.

They nicely have Tweet and Facebook buttons to share downtime, but no links to the service they are actually checking.

Embedly’s API is one of these services, however, their check is not configured correctly. Verelo does not include an API key in the URL, so when they hit our usage limit for unauthenticated requests we throw errors. If you rely on Verlo we have been down for most of the last 2 days.

Funny enough, this is their API call:

http://api.embedly.com/1/oembed?&maxwidth=400&maxheight=300&urls=https://t.co/04SmeCDl

That t.co url redirects to a reddit post for their own press release: https://t.co/04SmeCDl. Zero comments, whomp whomp.

I’m all for trying to to acquire new business, but by publicly shaming people via whatsdowntoday.com is down right dirty.

Attracting customers starts with building relationships, not alienating your target market. Your viral growth hack has turned into spam, time to go back to the drawing board.

If you would like to see Embedly’s actual status, you can check out Pingdom: http://stats.pingdom.com/q1pr2ct042w3

Ember at Embedly: Templates and the Router

screeley — Fri, 05 Apr 2013 14:19:00 -0400

This is the second post in our series on Ember. We will be building off the development environment we set up in the first post.

In practice, everything is based off the router and templates. Ember handles the dynamic creation of views and controllers. You really don’t need them until you start dealing with data and events.

By default Ember sets up two routes for you: application and index. application is the container for your entire application while index is the route for the root of your site ‘/’. You can create the two templates for these routes in the templates directory.

$ touch app/templates/application.handlebars
$ touch app/templates/index.handlebars

And for good measure we will create a global nav.

$ touch app/templates/nav.handlebars

The application template is a good place to create scaffolding; things that need to be on every page and also inside the scope of the application. Here is our simple application.handlebars:

{{ render "nav" }}
<div class="row">
 <div class="large-12 columns">
  {{outlet}}
 </div>
</div>

We just introduced two handlebars helpers that Ember uses: render and outlet. Handlebars helpers are a part of the Handlebars template language that is included with Ember. The render and outlet helpers are added by Ember.

About render from the docs:

Renders the named template in the current context using the singleton instance of the same-named controller.

render allows you to include one template into another. It’s much like the template helper, but also allows you to pass data along. In this case you could use either.

outlet is a placeholder that allows you to specify where the child node’s content will be rendered. They are much like {% block %} tags in Django or <%= yield ==%> in ERB.

Since application.handlebars is the base route the content from index.handlebars will be rendered where this outlet is. We can create a simple index.handlebars to prove this.

$ echo "<h1>Welcome To My App</h1>" > app/templates/index.handlebars

Routes

Routes are URL paths. To add a new page, you need to add a new route. So to add our about page, we need to add a route to our router.

App.Router.map(function(){
  this.route('about');
});

Ember is creating AboutRoute, AboutView, and AboutController. We don’t really need those for this exercise, but we do need a template with some content.

$ echo "<h1>About To My App</h1>" > app/templates/about.handlebars

If you visit ‘/#/about’ you will see the content.

Resources

When you want to start grouping routes, i.e. “/company/contact” and “/company/team” resources come in to play. The benefit here is that they share common templates. While you could so something like this:

App.Router.map(function(){
  this.route('contact', {path:'/company/contact'});
  this.route('team', {path:'/company/team'});
});

It becomes a mess really quickly. Instead you should use a resource.

App.Router.map(function(){
  this.resource('company', function(){
    this.route('contact');
    this.route('team');
  });
});

It is however a little more confusing when it comes to templates. Like the application route, we need to create a base container for all the leaf routes. This could be as simple as:

$ echo "{{outlet}}" > app/templates/company.handlebars

In this case, everything will just be added to the outlet in the application.handlebars. If you want every company page to have common display attributes, you can change the template:

<h1>Company</h1>
<div class="row">
  <div class="large-9 columns">
    {{outlet}}
  </div>
  <div id="sidebar" class="large-3 columns">
    {{outlet sidebar}}
  </div>
</div>

You’ll notice here that we added a named outlet. By default all content from the leaf routes will go into the {{outlet}} unless we tell it to go somewhere else instead.

We can then create the leaf templates.

$ mkdir app/templates/company
$ touch app/templates/company/index.html
$ touch app/templates/company/contact.html
$ touch app/templates/company/team.html

This is one of the nice things about having ember_templates set up and compiling templates for you. Ember expects the templates to be put into EM.TEMPLATES hash as ‘resource/route’ or you need to declare the templateName on the view. The former is less code.

Custom Routes

We set up that outlet earlier, it’s time that we use it. In order to use named outlets we need to modify the Routes behavior. Here we will change that Company Team Route to add a sidebar. In the main.js file we are going to add:

App.CompanyTeamRoute = Em.Route.extend({
  renderTemplate: function(controller, model){
    // Render the base template
    this._super(controller, model);
    // Render the bios template into the sidebar
    this.render('bios', {outlet: 'sidebar'});
  }
});

And then add the bios template:

$ echo "<h2>Bios</h2>" > app/templates/bios.handlebars

When you visit /#/company/team you will see the bios section in the template.

Alright, that was a quick primer into the Router and Templates, you can see the final code here: https://github.com/screeley/ember-demo-environment/tree/router. Next week, we’ll go into using View and Controllers.

Using Embedly Extract to Find Out What's Hot on Reddit

thejohnnest — Mon, 01 Apr 2013 14:51:00 -0400

Hello there embedders! You might have noticed that we have recently released a couple new products — today we’re going to talk about a couple features of the new /1/extract endpoint. Extract is great when you want to not only embed a URL but get a deeper understanding of its contents.

Let’s write up a quick application that scrapes the front page of any given subreddit and uses the extract endpoint to see what people are talking about by totaling all keywords and entities. Keywords are probably what you expect them to be, while entities are essentially just proper nouns. Note that you’ll need an Embedly API key to run this demo — you can get one for free if you don’t already have one!

First thing’s first: let’s grab the frontpage of a subreddit using reddit’s awesome JSON API. This is just a simple HTTP request to the subreddit frontpage with .json appended to the URL. We also define a function to extract all of the URLs from this blob:

import sys
import json
import urllib

import requests

def reddit(sub):
    """
    Retrieve the json representation of the given subreddit
    """
    url = 'http://reddit.com/r/%s.json' % sub
    resp = requests.get(url)
    return json.loads(resp.text)

def reddit_urls(js):
    """
    Return all the URLs in the given json blob of reddit data
    """
    return [x['data']['url'] for x in js['data']['children']]

The above code depends on one external package: the excellent requests package, which makes performing HTTP requests very simple.

Now that we have a list of URLs, let’s call embedly’s extract endpoint. For URLs that have any significant amount of text this response will include keywords and entites that summarize the text (take a look at an example response if you’d like to see what else is returned).

def embedly(url, key):
    """
    Call the embedly API and return the json blob. Returns None if there
    is an error processing the URL (e.g., HTTP 404)
    """
    quoted = urllib.quote(url)
    api = 'http://api.embed.ly/1/extract?url=%s&key=%s' % (quoted, key)
    resp = requests.get(api)
    if resp.status_code != 200:
        sys.stderr.write('Failed to process URL %s :(\n' % url)
        return None
    return json.loads(resp.text)

Okay, now we can just loop over all of the responses and sum together all of the counts and take the, say, ten highest counts. The code below operates on entities, but by changing ‘entities’ to ‘keywords’ in the line above sum_counts you can have it look at keywords instead!

def sum_counts(item_lists):
    """
    Takes a list of lists of items and mashes them all together into a dict
    mapping item_name -> item_score. The input items are either entity or
    keyword lists, which look like {'name': 'Dennis', 'count': 10} or
    {'name': 'sports', 'score': 33}, respectively.
    """
    ret = {}
    for item_list in item_lists:
        for item in item_list:
            # get either score or count so we can work with both
            # keywords and entities
            name, count = item['name'], item.get('count', item.get('score', 0))
            ret[name] = ret.get(name, 0) + count
    return ret

def top_n(item_dict, n):
    by_count = sorted(item_dict.iteritems(), key=lambda x: x[1],
        reverse=True)
    return by_count[:n]


def main(subreddit, key):
    urls = reddit_urls(reddit(subreddit))
    embeds = [embedly(url, key) for url in urls]
    # change 'entities' to 'keywords' to check out the keyword extraction!
    items = [x['entities'] for x in embeds if x]
    all_counts = sum_counts(items)

    top = top_n(all_counts, 10)
    print '\n'.join('%s %s' % (x[0], x[1]) for x in top)


if __name__ == '__main__':
    if len(sys.argv) < 3:
        sys.stderr.write('Usage: demo.py <subreddit> <your-embedly-key>\n')
    else:
        main(sys.argv[1], sys.argv[2])

Let’s try it out on /r/sports!

$ python demo.py sports <your-embedly-key>
NCAA 13
Ware 3
Sharks 2
Daily Caller 2
Kevin Ware 2
Louisville 2
Cal 2
Taylor Branch 1
San Jose Sharks 1
National Collegiate Players Association 1

Here we can see that the NCAA is by far the most popular entity mentioned, which should be no surprise considering we’re in the height of the March Madness tournament. Kevin Ware also makes an appearance, no doubt due to his unfortunate injury in Sunday’s game.

How about /r/worldnews?

$ python demo.py worldnews <your-embedly-key>
North Korea 66
India 30
Pakistan 27
U.S. 21
China 19
UN 17
Bolkovac 16
United States 14
EU 13
Novartis 13

Looks like North Korea is big right now, which isn’t too surprising considering what’s been going on over there.

Go grab the source if you’d like to play around. There are all sorts of ways to extend this — extract also pulls image metadata, so maybe you could make an image wall sorted by keyword or entity of what’s going on right now. Or use the related article feature to curate content on the current hottest topic. Speaking of related articles, we’ll be showcasing that feature in an upcoming post!

Sign up for a free account to start using Extract and run this example, or check out the other features and demos available for Extract. Other thoughts? Continue the conversation on Twitter

In Support of John

screeley — Mon, 01 Apr 2013 09:43:00 -0400

This is John.

John works at Embedly and he seems like a normal guy. He likes ultimate Frisbee, a good joke and time with friends. He has 10 fingers and 10 toes, but John was born with a great affliction. From the second he was put on this earth he had to endure overwhelming odds against him.

You see when John was delivered the doctor held him up, as Rafiki did to Simba, and proclaimed “April Fools.” Yes, John was born on the 1st of April.

Gasp, I know. Let that sink in for a second.

Like millions of “Foolies”, as they call themselves, he was subjected to a childhood of bad jokes. “Your parents must of thought you were a joke…” “Here little Johnnie, I got you the TMNT action figure you wanted. Nope, Ha, April Fools.” “John, we got you a car for your Birthday. Ha, April Fools, here’s a Frisbee.” “John, you were adopted… April Fools!” The courage that it takes, I can’t imagine.

The Foolies are a fellowship. Once a year, while everyone is out pranking they meet at a bar. They drink raspberry beer and not a single giggle is heard. They sit in peace as the world burns around them.

We at Embedly support the Foolies. We will not be participating in what tech has deemed a day of bad jokes thought up by insensitive marketing people to show their company has a “fun side.” It hurts real people like John and we cannot condone such actions.

There will be no scratch and sniff embeds and you will not be hearing Rick Astley.

Instead we will be silent in solemn solace with those less fortunate than us. John, we will not be pranking anyone today, we love you, stay strong.

- Sean

Ember at Embedly: Introduction to Ember Development

screeley — Fri, 29 Mar 2013 11:47:00 -0400

Every time I see Ember on Hacker News it’s being trashed, like MongoDB trashed. It sucks for an open source project when the herd mentality kicks in and it becomes cool to hate. Sadly, I’m not one of the cool kids.

Ember is excellent. We use it extensively here at Embedly for our developer dashboard. While none of us here are JavaScript experts, Ember has made us incredibly productive.

We are going to do a series of posts on Ember at Embedly. Mostly tips and tricks that have made us successful. These are not aimed at the pro, but the amateur.

A lot goes in to baking a production-quality website, and Ember adds to that. As a result, there is a lot of boilerplate when it comes to setting up the development environment. Rest assured that once you fight through the boilerplate, you’ll be able to iterate at an impressive rate.

Okay, so you have a new awesome idea of what you are going to build, how do you get started?

Yeoman/Grunt

Grunt is the Javascript Task Runner. It makes the set up of common tools like jshint, compass, uglify and development servers dead simple. We use it in almost every project now and it has made development a whole lot easier.

Yeoman is a thin wrapper around Grunt that really just builds out a directory structure and a few files. One member of your team will use it once and never think about it again.

So, let’s get started.

$ mkdir demo
$ cd demo
$ npm install -g yo grunt-cli bower

At the time of writing, the ember generator for yeoman is hosed, so you can do what you wish with that information. We can just use the default webapp generator.

$ yo webapp
$ npm install & bower install

This will build a pretty simple directory structure.

  app /
    index.html
    scripts/
      main.js
    styles/
      main.css
  Gruntfile.js
  package.json
  test /

Yeoman sets up the project so it can immediately be used. Run grunt’s server to launch your new web app.

$ grunt server

Tada, a browser will open to http://localhost:9000/ and show a nice little Allo message.

Getting Ember

Bower is Twitter’s package management tool for the Web. Let’s get Ember installed using bower.

$ bower install ember

Bower will handle getting all the right dependencies and install them in a components directory. You will then need to add Ember and Handlebars to your index.html file.

$ vim app/index.html

You can put the following lines near the bottom but above main.js:

<script src="components/handlebars/handlebars.js"></script>
<script src="components/ember/ember.js"></script>

Grunt will live reload your browser. You now have all the dependencies for a simple Ember app. You can prove it by opening up the console and running Em.

To recap, that was about 5 commands to get Ember completely ready to go:

$ npm install -g yo grunt-cli bower
$ yo webapp
$ npm install && bower install
$ bower install ember
$ vim app/index.html

If you complain about how hard Ember is to set up you are using the wrong tools.

Templates

Here on out is optional, but will probably save you some headaches later down the road. We will set up grunt-ember-templates, an Application and the Router.

The grunt-ember-templates module pre-compiles all your templates on the fly and adds them to Em.Templates, which is where Ember keeps your templates so it can match them to your views, based on naming convention. More on this later.

$ npm install grunt-ember-templates --save-dev
# Create the templates directory.
$ mkdir app/templates
$ touch app/templates/index.handlebars

We are going to have to update the Gruntfile to run ember_templates

Grunt is configured by a Gruntfile.js, which uses JSON to define a bunch of different actions that Grunt can perform for you. This is where we’ll hook up grunt-ember-templates to Grunt.

$ vim Gruntfile.js

In the Gruntfile, you are going to load the task before, I generally put it before the grunt.initConfig

grunt.loadNpmTasks('grunt-ember-templates');

Inside the grunt.initConfig you will need to set up the compile task to point at all the templates.

ember_templates: {
  compile: {
    options: {
      templateName: function(sourceFile) {
        return sourceFile.replace(/app\/templates\//, '');
      }
    },
    files: {
      "<%= yeoman.app %>/scripts/templates.js": ["<%= yeoman.app %>/templates/**/*.handlebars"]
    }
  }
},

Next tell the watch command to look for changes in the templates directory to make the live reload work correctly:

watch {
  ...
  ember_templates: {
      files: '<%= yeoman.app %>/templates/**/*.handlebars',
      tasks: ['ember_templates']
  },
  ...
}

Lastly, add ember_templates to the server task, you are looking for this line:

grunt.registerTask('server', function (target) {
    ...
    grunt.task.run([
        ...
        'compass:server',
        'ember_templates', // Add this line.
        'livereload-start',
        ...
    ]);
});

Once you restart grunt server, you can take a gander at the app/scripts/templates.js file in scripts. It should look something like this:

Ember.TEMPLATES["index"] = Ember.Handlebars.template
...

Add that script to the index.html file to make sure the browser loads it.

<script src="scripts/templates.js"></script>

That was kind of a detour, you don’t really need to set up ember_templates, you can use inline script tags or Em.Handlebars.compile, but this method offers a nice separation of HTML and javascript. For a quick reference you can see the completed Gruntfile here.

Application

The last step here will be to set up a simple App and Router to take advantage of our setup.

You can add the main.js to the end of index.html and delete whatever is in that file.

<script src="scripts/main.js"></script>

Next set up a rootElement for Ember to use, this will make sure that Ember puts the html where you want it to. Replace the div with the class container with:

<div id="app"></div>

Next add a simple Application to main.js:

App = Em.Application.create({
  rootElement: $('#app'),
});

You can then edit index.handlebars with a simple h1

$ echo "<h1>My App</h1>" > app/templates/index.handlebars

When grunt reloads the server, your browser will have My App as the header. There is a little magic going on here. Ember is creating a Router for you and by default creates and Application Route and an Index Route. This is why just adding an index.handlebars works.

The Router is dead simple to set up as well. In main.js add

App.Router.map(function(){
  this.route('about');
});

And then create the about template

$ echo "<h1>About</h1>" > app/templates/about.handlebars

If you navigate to /#/about, boom, you are now in the About route. You can also link to the about page from index by adding:

{{#linkTo "about"}}About{{/linkTo}}

This all works based on naming conventions. You can create an AboutController, AboutRoute and AboutView, but Ember will take care of that for these simple routes. More on that later.

Recap

Setting up Ember is a lot of boiler plate code, and there are some really good tools out there that do it for you. Nothing I have described above is earth shattering, but it should get you set up for your next Ember project.

I put the complete demo on Github for reference you can find it at github.com/screeley/ember-demo-environment.

For our next post in the Ember series, we will go into Ember basics; Views, Routing and Outlets. Stay tuned.

- Sean

Discuss on Hacker News