blog.mashupguide.net Open in urlscan Pro
69.163.253.2  Public Scan

Form analysis
1 forms found in the DOM

GET https://blog.mashupguide.net

<form id="searchform" method="get" action="https://blog.mashupguide.net">
  <div>
    <input id="s" name="s" type="text" value="" size="10">
    <input id="searchsubmit" name="searchsubmit" type="submit" value="Search">
  </div>
</form>

Text Content

MASHUP GUIDE

A blog about Raymond Yee's Book Pro Web 2.0 Mashups: Remixing Data and Web
Services
Skip to content
 * About Mashup Guide (mashupguide.net)
 * Text of the Book (v 1.0)
 * Author Bio: Raymond Yee
 * Book
 * Things that are currently "out of scope" that would be nice to write about….
 * About the Technical Reviewer
 * Acknowledgements
 * How to download the code for the book
 * Friends


DECAY IN THE AMAZON APIS

In Chapter 17 of Pro Web 2.0 Mashups, I created a mashup of an Amazon Wishlist
and Google Spreadsheets. When I returned to examine my code last night, I
learned that it no longer worked.  Why?  First, the  Amazon Ecommerce API
morphed into the Amazon Product Advertising API; I was puzzled why the API
wasn't listed where I expected it to be.  Unfortunately, Amazon, in its
infinite and inscrutable wisdom,  also decided to kill the ListLookup
operation, the one call that I depended on to retrieve the content of my Amazon
wishlist.  (I'm not alone in having broken applications because of this
change.)

So what to do now?  Interestingly enough, someone just announced a JSON feed
service for a given wishlist, for example, Jeff Bezos' wishlist and mine (in
JSON).  I hope it stays around.  How does it work given the demise of the
ListLookup operation?  My guess is that some sort of screen-scraping is going
on.

2011 02 23 raymondyee Amazon
Amazon EC2
Chapter 17
screen scraping Comments (0) Permalink


MAPPER FOR SERIALIZING XML ELEMENTTREE WITH SQLALCHEMY

This morning I've been working out how to configure a SQLAlchemy mapper to
enable me to store an XML blob to relational databases.  Storing the XML
element is a bit of  hack right now; ultimately, I want to map all the relevant
pieces of the XML to appropriate Python object attributes.  But until I figure
that mapping out, I'm saving the XML so in theory I can avoid making yet another
API call.

I think I may have found relevant examples to guide me.  SQLAlchemy comes with
a set of examples:  see docs and code (v 0.6.1) — specifically, the examples
for serializing XML.  The first approach (using PickleType) is a simple and
expedient approach, a good enough one to start with.   I'll come back to study
the other ones.

Tagged Python, SQLAlchemy, XML
2010 07 01 raymondyee Uncategorized Comments (0) Permalink


LINKEDIN API: FIRST STEPS USING PYTHON

My enthusiasm  for LinkedIn increased dramatically once I learned that LinkedIn
had opened up its API to the public at large.  What is still unclear to me is
how much the API allows one to get data in and out LinkedIn.   One of the best
ways to find out:  dive in and see what we can learn.

In this post, I describe  some first steps you can take to learn to access the
LinkedIn API  with Python (a favorite programming language of mine and many):

 1. Get oriented by looking the main page for the API, which in this case, is
     LinkedIn Developer Network.
 2. You'll need a set of developer keys for each application, which you can get
    by registering an application.  You'll be asked to login with your LinkedIn
    user account email/password.  If you don't yet have a LinkedIn user
    account, sign up for one. (Using the API doesn't require a separate
    developer account.)
 3. I found a few tricky bits in the registration process.   First of all,
    you're going to have remind yourself (or teach yourself for the first time)
    the basics of OAuth, an open protocol used by LinkedIn (and other websites)
    to authorize users.   (I won't attempt to provide such a tutorial here. A
    set of slides  from LinkedIn does a pretty good job of giving an overview
    of OAuth. ) The second tricky part was that I forgot whether OAuth could
    support desktop applications.  It turns out that you do get 3 options for
    the types of app you are registering: desktop, web, and mobile.  In my
    case, I registered that I was creating a desktop app.
 4. When you are finished registering your application, you will get two
    important parameters for your app:  the OAuth (consumer) key and secret.
    You will need these two parameters for your Python application.
 5. You can choose to work with the API directly at  the HTTP level protocol
    level or look for API libraries that wrap the protocol.  I searched for
    such a Python library, and found pylinkedin (a barebone but functional
    library), whose source you can get via mercurial by hg clone
    https://pylinkedin.googlecode.com/hg/ pylinkedin
 6. Install the library using the usual python setup.py install.  pylinkedin
    was tested on Python 2.5 but so far, I've found it to work on Python 2.6.
    Note that the library requires that you have the  oauth Python library
    installed (available via svn at
     http://oauth.googlecode.com/svn/code/python/
 7. You will need to correct a small bug that I found in pylinkedin (around the
    parsing of companies) — you'll need to edit the __init__.py file according
    to instructions I posted.

Now we're ready to run the following code, which will display the first and last
name of your LinkedIn "connections" (the people in your immediate circle).
 Remember that key and secret you got when you registered your application:
plug them into the following program.  (Note that a browser window should open
prompting you for your LinkedIn password.  You'll then have to enter a code
that LinkedIn gives you to let the program access your LinkedIn data.)

# insert your application KEY and SECRET

API_KEY = "_FILL_IN_YOUR_KEY_"
SECRET_KEY = "_FILL_IN_YOUR_SECRET_"

import webbrowser

from linkedin import LinkedIn
li = LinkedIn(API_KEY, SECRET_KEY)

token = li.getRequestToken(None)

# prompt user in the web browser to login to LinkedIn and then enter a code that LinkedIn gives to the user

auth_url = li.getAuthorizeUrl(token)
webbrowser.open(auth_url)
validator = input("Enter token: ")
access_token=li.getAccessToken(token,validator)

# list all connections

connections = li.connections_api.getMyConnections(access_token)

print "number of connections: ", len(connections)
for c in connections:
    print c.firstname, " ", c.lastname

There are obviously improvements to make in this starter code — but this should
get us all on our way.

Acknowledgements: Thanks to LinkedIn for the API and to Max Lynch for
pylinkedin, which made getting into the API much easier!

Tagged LinkedIn API, pylinkedin, Python
2010 02 05 raymond.yee Uncategorized Comments (0) Permalink


ZOTERO REST API: EARLY DEVELOPMENTS

Readers of my book know that I'm an avid user of social bookmarking and online
bibliographic systems. As  a big fan of Zotero ("a free, easy-to-use Firefox
extension to help you collect, manage, and cite your research sources"),  I
have been looking for  ways to further integrate Zotero (both the client and
the web front end) into my workflows.  Specifically, I would like to experiment
with integrating Zotero into my teaching this semester,  specifically to help
my students and I  share references.

I've played with scripting Zotero from within the web browser using
Chickenfoot.  However, a fully fleshed out Zotero REST API will, I think, be
even helpful in tying Zotero to other applications.  There is a Zotero REST API
under development, though it is currently limited and not well publicized.

The best publicly available documentation of the API as it stands that I know of
is Jeremy Boggs' code phpZotero.   Reading it taught me how to make some basic
calls.

(Note: I currently have two separate Zotero accounts:  a personal one (rdhyee)
and one for my course (mixingandremixinginfo).  Ideally, I'd rather manage only
one account but have two because, from what I understand, there is no
fine-grained privacy control that would let me  keep some references private
while leaving most public.    Moreover, it seems that the REST API can read
only public items for now — so I'll be applying it to mixingandremixinginfo for
now. )

Let me write out what you can do with the API to get started.

 1. Create a key at  Zotero | Settings > Feeds/API (Though I've used a key, I
    don't see any indication of key usage in my panel — is this a bug?)
 2. Go to the Zotero privacy tab to check "Publish Entire Library" and "Publish
    Notes" to make the items (and optionally notes) visible to your API key.
 3. Figure out the userID for the account.  You can do so but doing a HTTP GET
    on
    http://www.zotero.org/api/users/{username} e.g.,
    http://www.zotero.org/api/users/mixingandremixinginfo — see
    
    
    
    <?xml version="1.0" encoding="utf-8"?>
    <entry xmlns="http://www.w3.org/2005/Atom" xmlns:zapi="http://zotero.org/ns/api">
        <title>mixingandremixinginfo</title>
        <author>
            <name>Zotero</name>
            <uri>http://zotero.org</uri>
        </author>
        <id>http://zotero.org/users/119961</id>
        <published>2010-01-26T22:43:38-05:00</published>
        <updated>2010-01-26T22:46:11-05:00</updated>
        <link rel="self" type="application/atom+xml"
            href="http://api.zotero.org/users/119961"/>
        <link rel="alternate" type="text/html"
            href="http://zotero.org/mixingandremixinginfo/119961"/>
        <content type="application/xml">
            <zapi:userslug>mixingandremixinginfo</zapi:userslug>
            <zapi:profile>
                <zapi:affiliation><![CDATA[UC Berkeley]]></zapi:affiliation>
                <zapi:bio><![CDATA[<p>This is a public account associated with the Mixing and Remixing Information course at the UC Berkeley School of Information</p>]]></zapi:bio>
                <zapi:disciplines><![CDATA[Information Science and Technology]]></zapi:disciplines>
                <zapi:location><![CDATA[Berkeley, CA]]></zapi:location>
                <zapi:realname><![CDATA[Mixing and Remixing Information]]></zapi:realname>
                <zapi:website><![CDATA[http://blog.mixingandremixing.info]]></zapi:website>
            </zapi:profile>
        </content>
    </entry>
    
    
    and then parse out for the id element   (e.g., 119961)

 4. Now to get items and collections, note that the base URL for the API
    ishttps://api.zotero.org/
    and that you can use two
    parameters:  username,  apiKey in
    curl -v -k -X GET
    "https://api.zotero.org/users/119961/items?username={username}&apiKey={apiKey}"
    to return publicly visible items.
 5. You can get data about a  specific item
    (e.g.,http://www.zotero.org/mixingandremixinginfo/items/102472156)with
    
    curl -v -k -X GET
    "https://api.zotero.org/users/{userid}/items/{itemid}?username={username}&apiKey={apiKey}"
    
    e.g., curl -v -k -X GET
    "https://api.zotero.org/users/119961/item/102472156?username=mixingandremixinginfo&apiKey={apiKey}"

 6. Jeremy's phpCode hints at other things you can read from the API, including
    user collections and their  items, that I've not tried accessing.  However
    doing so should be straightforward.

Don't consider what I've written here as definitive by any means since  I don't
know of any publicly available official documentation.   I'm eager to see the
further development of the API, including the ability to access more data (about
users,  groups,  etc) and the ability to write to Zotero servers.

Comments welcome!

Tagged Zotero
2010 01 28 raymond.yee Chapter 14 Comments (0) Permalink


UPDATES TO CHAPTER 1

Some updates I'm making to Chapter 1

 1. Re: LibraryLookup Bookmarklet —
    http://weblog.infoworld.com/udell/stories/2002/12/11/librarylookup.html ->
    http://jonudell.net/LibraryLookupGenerator.html .   The LibraryLookup
    Project is a good page to use as a reference for the project.
 2. http://weblog.infoworld.com/udell/2006/01/30.html ->
    http://web.archive.org/web/20080616122712/http://weblog.infoworld.com/udell/2006/01/30.html
    (thanks to archive.org)

2010 01 25 raymond.yee Chapter 01: Learning from Specific Mashups Comments (0)
Permalink


CREATING BOOK XHTML FROM DOCBOOK

Ever since Pro Web 2.0 Mashups came out, I've wanted to get the book on the
web.  Publishing PDFs was a start — but I have envisioned developing a
full-blown web application, a book that would could interact with my readers, be
self-correcting and self-updating.  It's only appropriate that a book about
APIs and mashups should itself embody the techniques that are describe in it!

Well, it's going to be a while until I get there — but I'm happy that I haven't
taken the next step:  publishing a (X)HTML version of the book.   The
canonical version of the book ended up being a series of QuarkXPress files.  I
had written some Python appscript programs to convert the book to a simple
homebrew XML representation but didn't have sufficient time to take it all the
way to DocBook.  I hired Liza Daly's Threepress Consulting to do the bulk of
the conversion to DocBook, leaving me some labor intensive details to fit my
budget.  (BTW, Liza is a great person to work with, very  smart and responsive
to my queries. )

With the book in DocBook, I have been using oXygen 11 to edit the files and
transform them into XHTML. I was hunting around for CSS files for
DocBook-derived XHTML, but found surprisingly few options.  The one I'm
currently settling on to get me started is the stylesheet  for FreeBSD
documentation (see the style in action on the FreeBSD site).   I'll definitely
want to customize the stylesheet for the book to reflect the look and feel I
desire — but I'm happy with the starting point.

Tagged CSS, DocBook
2010 01 21 raymond.yee Meta Comments (0) Permalink


A ROUGH XHTML VERSION OF MY BOOK IS NOW LIVE

I'm pleased to announce that I've posted an  XHTML version of Pro Web 2.0
Mashups:  Remixing Data and Web Services. This should make it much easier for
folks to use my book.  It also opens up some good opportunities for me to
update the book, which I plan to do as I teach my Mixing and Remixing
Information course this semester.

A warning: the XHTML translation has some technical flaws that I'm working to
fix.  However, I figured that it was better to put a working version up and
then fix it along the way.

Enjoy and let me know what you think!

Tagged XHTML
2010 01 20 raymond.yee Meta Comments (0) Permalink


GOOGLE MASHUP EDITOR WILL BE SHUT DOWN

I'm sad about the announcement that Google is shutting down its Mashup Editor —
one reason being that I had spent a fair amount of effort writing about it in
Chapter 11 of  my book.  Oh well.  The Google App Engine is touted as a
suitable and more powerful alternative to the Mashup Editor — but I have to
agree with that the comment that the simplicity of the Mashup Editor was a
virtue.

Tagged Google, Google Mashup Editor
2009 01 18 raymond.yee Chapter 11
Google Comments (0) Permalink


SERVICES BUILT UPON AMAZON EC2

According to How To: Getting Started with Amazon EC2 – PaulStamatiou.com, the
following services are built on top of EC2 (and thereby perhaps make scaling up
EC2 easier):

 * Web-based Cloud Computing Management Platform by RightScale
 * scalr – Google Code
 * WeoCEO – Easy to Use. EC2 Love

What other ones are out there?

2008 12 18 raymond.yee Amazon
Amazon EC2 Comments (0) Permalink


S3AJAX: CREATING BUCKETS AND UPLOADING KEYS

Continuing on Mashup Guide :: listing keys with S3Ajax, here I present a
Chickenfoot script to create a bucket and upload a file (specifically, it
creates a bucket by the name of raymondyeetest and uploads a file
(D:\Document\PersonalInfoRemixBook\examples\ch16\Exported Items.rdf from my
WinXP machine) to the key exportitems.rdf in the bucket:

include("D:\\document\\JavaScriptLib\\sha1.js");
include("D:\\document\\JavaScriptLib\\S3Ajax.js");

var AWSAccessKeyId = "[AWSAccessKeyId]";
var AWSSecretAccessKey = "[AWSSecretAccessKey]";

S3Ajax.URL = 'http://s3.amazonaws.com';
S3Ajax.DEBUG = true;
S3Ajax.KEY_ID = AWSAccessKeyId;
S3Ajax.SECRET_KEY = AWSSecretAccessKey;

// function to read contents from a file

function read_file_contents(aFileURL) {
  // https://developer.mozilla.org/en/Code_snippets/File_I%2f%2fO#Binary_File

  var ios = Components.classes["@mozilla.org/network/io-service;1"]
                      .getService(Components.interfaces.nsIIOService);
  var url = ios.newURI(aFileURL, null, null);

  if (!url || !url.schemeIs("file")) throw "Expected a file URL.";
  var bFile = url.QueryInterface(Components.interfaces.nsIFileURL).file;
  var istream = Components.classes["@mozilla.org/network/file-input-stream;1"]
                          .createInstance(Components.interfaces.nsIFileInputStream);
  istream.init(bFile, -1, -1, false);
  var bstream = Components.classes["@mozilla.org/binaryinputstream;1"]
                          .createInstance(Components.interfaces.nsIBinaryInputStream);
  bstream.setInputStream(istream);
  var bytes = bstream.readBytes(bstream.available());
  return bytes;

}

// create a bucket

var newBucketName = 'raymondyeetest';

S3Ajax.createBucket(newBucketName, function() {
    output("created a buicket: " + newBucketName);
    s3_list();
}, function () {
      output ("error in createBucket");
   }
);

// add a key to a bucket

var fileURL = 'file:///D:/Document/PersonalInfoRemixBook/examples/ch16/Exported%20Items.rdf'
var content = read_file_contents(fileURL);
alert(content);

S3Ajax.put(newBucketName , "exportitems.rdf", content,
    {
       content_type: "application/xml; charset=UTF-8",
       meta:  {'creator':'Raymond Yee'},
       acl:    "public-read",
    },
    function(req) {
        output("Upload succeeded");
    },
    function(req, obj) {
        output("Upload failed");
    }
);

A few things to note about this code:

 * read_file_contents() doesn't strike me as a terribly elegant way of reading
   contents from a file — but that's what I have working for now
 * a tricky part of getting this to work was to note that in FF 3.x, 
   charset=UTF-8 is automatically tacked on to content-type HTTP request header
   in a xmlhttprequest — I don't know how to change the charset or whether you
   can — and why UTF-8 is being tacked on.  But figuring out the charset was
   crucial to getting this working since the content-type HTTP header is used in
   the calculation of the Amazon S3 signature.

Tagged Amazon S3
2008 12 17 raymond.yee AJAX
Amazon S3
chickenfoot
javascript Comments (0) Permalink
« Older posts



 * PAGES
   
   * About Mashup Guide (mashupguide.net)
   * Text of the Book (v 1.0)
   * Author Bio: Raymond Yee
   * Book
   * Things that are currently "out of scope" that would be nice to write
     about….
   * About the Technical Reviewer
   * Acknowledgements
   * How to download the code for the book
   * Friends


 * CATEGORIES
   
   * AJAX
   * Amazon
   * Amazon EC2
   * Amazon S3
   * Apress
   * BackgroundReading
   * Chapter 01: Learning from Specific Mashups
   * Chapter 02
   * Chapter 03
   * Chapter 04
   * Chapter 05
   * Chapter 06
   * Chapter 07
   * Chapter 08
   * Chapter 09
   * Chapter 10
   * Chapter 11
   * Chapter 13
   * Chapter 14
   * Chapter 15
   * Chapter 16
   * Chapter 17
   * Chapter 18
   * Chapter 19
   * chickenfoot
   * Creative Commons
   * del.icio.us
   * design patterns
   * drafts
   * email
   * Firefox
   * Flickr
   * geocoding
   * Google
   * Google AJAX Libraries API
   * Google Earth
   * Google Maps
   * Google Reader
   * Introduction
   * ISBN
   * javascript
   * KML
   * LibraryLookup
   * maps
   * mashup
   * mashup camp
   * Meta
   * Microsoft
   * Notelets
   * office suites
   * open content
   * ProgrammableWeb
   * Publishing
   * remix
   * research
   * REST
   * Riya
   * RSS
   * screen scraping
   * search
   * SOAP
   * storage
   * tagging
   * teaching
   * technorati
   * Thunderbird
   * Uncategorized
   * web browser
   * web services
   * weblogging
   * Wikipedia
   * WordPress
   * WSDL
   * XML
   * Yahoo
   * Yahoo! Pipes


 * TAGS
   
   AJAX Amazon S3 blurb cover book chickenfoot CSS del.icio.us DocBook Google
   Google Mashup Editor interview javascript libraries Library of Congress
   LinkedIn API mashup camp name change OpenAjax OpenID pylinkedin Python
   QuarkXPress publishing reviews S3 S3Ajax SQLAlchemy WordPress XHTML XML
   Yahoo! Pipes Zotero


 * BLOGROLL
   
   * Programmable Web


 * RSS FEEDS
   
   * All posts
   * All comments


 * META
   
   * Log in


 * BLOG SEARCH
   
   

© 2024 Raymond Yee | Thanks, WordPress | Barthelme theme by Scott Allan Wallick
| Standards Compliant XHTML & CSS | RSS Posts & Comments