I have reached a settlement with Sky.

tl;dr

Sky News stole my copyrighted work and distributed it without credit or payment.

I asked them to pay £1,500.

They refused.

Full Story

During the recent O2 brouhaha I recorded a video showing how the issue could affect people. I deliberately gave it the standard YouTube licence rather than the Creative Commons licence.

Later that evening, I was alerted to the fact that Sky News had broadcast my video without first seeking permission.

Just did Sky News spot about the @O2 controversy. From sound of what came down my earpiece they also used @edent's vid https://t.co/tBr2SoAp

25/01/2012 17:41 via EchofonReplyRetweetFavorite

@gcluley

Graham Cluley

I grumbled a bit on Twitter and was contacted by a representative of Sky News. He was very quick in ascertaining that my video was used without my permission and arranged to prevent it being shown again.

I was not happy about this state of affairs. BSkyB and News International have been very vocal about digital piracy and copyright infringement. They have relentlessly pursued a copyright maximalist agenda which – I believe – is damaging to the creative industries. Not to mention rampantly hypocritical.

Sky News accept user generated videos, with this stern warning:

Copyright protects the interests of the people and companies who create these products. If the content or product or marks in your video are owned by someone else, you are infringing copyright and run the risk being prosecuted.

It is important that you understand that you cannot take other people’s creations and use them as you see fit.

Uploading somebody else’s video is no different to taking something from a shop without paying. Copyright infringement damages the music, film and television businesses and the future development of music, film and TV programmes.

From Sky News’ Frequently Asked Questions about user videos. Emphasis added.

Sky News have a history of ignoring copyright and infringing the moral rights of authors.

Unfortunately, the Digital Economy Act doesn’t allow me to sue Sky News for distributing my content for free without my permission. An individual can lose their Internet access for sharing a movie, however there don’t seem to be any sanctions against a large company for sharing my copyrighted work without permission.

I don’t have the resources to fight a legal battle against Sky. So I decided to settle for cold, hard cash.

Originally, Sky made the following offer:

For a short youtube video like that we would normally pay around £50 & would be more than willing to pay an extra £25 for the fact you weren’t asked in the inconvenience it has caused you.

The NUJ publish a freelance rate card. The rate for freelance video should be around £300 per minute.

I farted in Sky’s general direction. I wanted £300 for the broadcast of the video, plus £1,200. I calculated that as £400 for them failing to ask permission, another £400 for them infringing my copyright, and then £400 for them violating my moral rights.

Even if they didn’t agree with the above reasoning, it is usual to charge four times time standard licence rate when a copyright owner is asked to assign all rights away. Effectively, Sky News had unilaterally assigned all my rights to them – so I felt justified asking for this sum.

Overall, I thought £1,500 was reasonable – especially when you consider that the Daily Mail paid £2,000 for using photographs without permission.

Sky came back with an offer of £300.

I pointed out that s107 of the Copyright, Designs and Patents Act 1988 had provisions for fines of up to £50,000, or a six month prison sentence. In comparison £1,500 seems modest.

Sky replied:

As I indicated, we do not yet agree on a sensible figure for this use.

Bearing in mind you are now invoking the Copyright Designs and Patents Act I have placed this matter in the hands of our lawyers.

This does not represent an unwillingness to come to an agreement between us but, unfortunately, it is likely to slow the progress slightly.

Now I’m waiting to hear back from their lawyers. I wonder what their hourly rate is?

I’ll update this post when I know more…

A Settlement

I have accepted an offer of £300. A few minutes ago, I received this email from Sky:

After consulting with our Sky lawyers our position is that we believe a £300 settlement is a fair and appropriate sum.
Our position is:

• The £300 is in respect of what you describes as “infringement of copyright” rather than any “union rate”;
• Contrary to what you claim, we did not act as if you had assigned us all rights. Specifically, we did not claim ownership nor seek to profit from it by licensing to others;
• Criminal liability will not attach in relation to an inadvertent use of footage;
• English law does not recognise violation of moral rights;
• There is no authority that an infringement in these circumstances attracts four times the usual licence fee. To the contrary, the usual measure is what the reasonable cost of licensing would have been.
• Our offer is generous for the reasons above and we will not increase it.

May I also stress that when you are relating this issue to third parties on whatever platform I would consider it unfair if you did not relay the fact that we immediately acknowledged your copyright and sought to bring redress. I stress, once again, that we take copyright and its infringement very seriously at Sky News.

I’m not a copyright lawyer, so am in no position to argue against their formidable legal might. I have invoiced them for £300. I must point out that – once I contacted them – Sky were very quick to take down the infringing content and have been unfailingly polite in their dealings with me.

I will be donating £100 to both The Open Rights Group and MySociety. And drinking the rest.

Thank you for all your comments, tweets, and messages of support.

Footnote

I’m currently on holiday – so comment moderation and updates may be delayed.

• Verizon Droid
• Stock Market
• Hotels

Errr…..? What? If you’re following me on Twitter for any of the above, perhaps you’d like to consider your life choices.

So, I opted-out of Klout.

Their opt-out page is full of feel-good nonsense to try and convince people not to remove their own profile. It didn’t work on me. They asked for my reasons for leaving, and this is what I told them:

Three reasons.
1) You think I’m influential about things that I’ve never tweeted about (Stock market? Verizon?)
2) I didn’t ask to sign up.
3) It’s taken you too long to offer this opt-out. If I’d known from the start I could have opted out, I would have been fairly relaxed. The fact that you’ve resisted for so long to offer the opt-out makes me think you’re not trustworthy.

I am indebted to LoudMouthMan for taking Klout to task. I’m sure that without his persistence, Klout would still be peddling lies about me.

For a good overview of why “influence measument” site like Klout don’t really work, read The Fallacy of Presumed Influence by Jonathan Macdonald.

Twitter’s way of linking URLs is broken. It’s annoying to users, and a pain in the arse to developers. This quick post talks about the problem and offers a solution.

I’ve raised a bug with Twitter and I hope you’ll star it as important to you.

Preamble

A common trope in programming classes is “how do you detect valid email address?”

It should be obvious, right? A string of text, an @, a domain – probably ending in .com.
As it turns out, it’s not that simple. “who+o’toole@invalid.museum” is a potentially valid address, for example.
There are literally thousands of ways to detect the potentially infinite variety of email addresses.

The same is true for URLs – and slavish adherence to guidelines is killing Twitter’s usefulness.

The URL Matching Problem

Which of these strings should be turned into hyperlinks?

www.bbc.co.uk

example.com

http://test

https://test.test

ftp://news.com

As it happens, Twitter only matches “https://test.test” and none of the others.

Twitter’s matching regex is, as far as I can tell, this

If it starts with http:// or https:// and has a dot in it - it's a URL

I think this is a serious weakness. Twitter users are sharing URLs which their followers can’t click on – Twitter is also linking to URLs which don’t exist.

I’ve picked these examples more or less at random.

Solution?

Much like the email regexes, I would take a much more lax approach. Essentially, if it looks vaguely like a URL – link to it.

I would suggest the following rules:

• If it starts with a protocol – http:// ftp:// tel: etc – create a hyperlink.
• If it starts with www. – create a hyperlink.
• If it ends . then a valid TLD – create a hyperlink.
• If it contains a valid TLD followed by a slash then some other characters – create a hyperlink.

The “correct” method would then be for Twitter to perform an HTTP HEAD request to see if the URL is potentially valid. There are three drawbacks to this.

1. It may place excessive load on Twitter’s servers to process and cache these requests.
2. The URL may be that of an Intranet site – and thus inaccessible to Twitter.
3. The URL may be valid but temporarily inaccessible.

Regardless of the method, surely it’s inexcusable that “www.example.com” isn’t detected as a URL whereas “http://bork.bork.bork” is?

ACTION!

If you think Twitter’s approach to hyperlinks is wrong – please make your voice heard at the bug report.

Wouldn’t it make sense to use a QR code as well? That way people could quickly scan, and be taken straight to the discussion, rather than have to fire up Twitter and do a manual search.

As it happens, it’s slightly tricky to make a QR code which searches for a Twitter hashtag.

There are two things to note:

1. Twitter’s search URLs are annoyingly different from every other search URL on the planet.
2. You will need to take care of URL Encoding for special characters.

Building The Search Query

A typical Twitter search URL is

https://mobile.twitter.com/#!/search/

With the query at the end, so a search for “Transformers” would be

https://mobile.twitter.com/#!/search/Transformers

Encoding Correctly

Hash symbols (#) need to be URL Encoded. In this case, the hash becomes “%23″

https://mobile.twitter.com/#!/search/%23Transformers

However, there is a “gotcha”. Because of the… special… way Twitter constructs its search URLs, we have to URL Encode the percentage sign! So, the “%” becomes “%25″.

So, our final URL becomes

https://mobile.twitter.com/#!/search/%2523Transformers

Finally, if you want to use something like Google Charts to create your QR codes, you’ll need the URL Encode all of the : / # ! and other special characters.

https://chart.googleapis.com/chart?chs=200x200&cht=qr&chl=
   https%3A%2F%2Fmobile.twitter.com%2F%23%21%2Fsearch%2F%2523Transformers

The Final Result

We end up with a QR code which can be scanned to take the user directly to the hashtag they are searching for,

As you've discovered I will become @ITVLauraK in September! Thanks for all the lovely tweets - Back in Westminster tomorrow

22/06/2011 13:31 via webReplyRetweetFavorite

@BBCLauraK

Laura Kuenssberg

Which got me wondering.

Why doesn't @BBCLauraK simply rename her account to @ITVLauraK in September? Or are her followers the BBC's property?

22/06/2011 13:48 via Shkspr's SSL DabrReplyRetweetFavorite

@edent

Terence Eden

Twitter allows any user to change their name and keep all their followers. All of Laura’s followers, friends, favourites, tweets, retweets, and lists will dissapear when she swaps to a new account.

Or will they? Is @BBCLauraK the twitter account for the BBC’s Chief Political Correspondent? Or is it Laura’s account?
If the former, I’d expect it to be taken over by her successor and renamed @BBCgordonGopher (or whoever).
If the latter, it makes sense for her to rename the account to @ITVlauraK (or whatever).

Looking at her feed, it’s broadcast only with almost no “personal” tweets. Certainly no “Urgh, hungover” or “Trains late again!” tweets.

So, it probably makes sense for Laura to hand over control of the account when she leaves.

Impact On You

How much work tweeting do you do from your Twitter account? If you left your job, would your bosses be able to demand you surrender the account to them?

Sure, if you’re tweeting from @NameOfYourCompany – it’s their account. But if you’re @CompanyName_Bob – do you get to keep it when you go? What if you’re just @You – and you tweet about the company you work for – what happens to all that good-will when you go?

Work Phone vs Work Email

When I first joined Vodafone nearly a decade ago, I was given a mobile phone number. Our office was quite revolutionary because there we no desk phones! Of course not; we were in the mobile phone business.

This confused some people greatly. I’d give them my number and they’d say “yes, but how can I reach you at your desk?”
And I’d say “Errrr…. I always have my mobile with me. Assuming it’s me you want to reach, ring that number. Why would you want to talk to someone who happened to be sitting near where I work?”

Since then, I’ve kept the same mobile number. If you want to reach me – that’s the number you call. No matter who I end up working for, that’s my number. When I move roles, I take my number with me. When I leave a job, I take my number with me.

Yet I don’t take my email address with me. Mails to terence.eden@vodafone.com will go unanswered – I hope!
I can understand that – I don’t want confidential documents forwarded to me. The domain name would make it look like I was still affiliated with an old employer.

But my phone number comes to me, Terence Eden. Wherever I am. Whoever I work for.

My Twitter account – @edent – is personal. It’s mine. No boss can take it away from me.

What about yours?

Rather than using an external service like Embed.ly to retrieve thumbnails, all the data is embedded within Twitter Entities.

So, if you request a status using “include_entities=true“, you will be able to grab the image and display the thumbnail using the following code.

So, a tweet like this:

#Photos on Twitter: taking flight http://t.co/qbJx26r

02/06/2011 18:53 via webReplyRetweetFavorite

@twitter

Twitter

Notes

This is very rough and ready proof of concept code. Beware of the following:

• This will only take the first image from the tweet.
• Only images are supported – I’m not sure how their proposed video sharing will work.
• There’s no error checking.
• In the above code, the https URL is used – if you want a non-SSL link, you’ll need to remove the “_https”

Enjoy!

Twitter’s secure API hides the contents of the tweets you are reading. But it doesn’t hide the images of those you converse with.

Raised as Issue 2175.

A Bit More Detail

Twitter has a secure (HTTPS) and insecure (HTTP) API.

When calling the secure API, all the content of the returned message (tweets) are encrypted. Eavesdroppers only see the cipher-text – essentially garbage.

However, within that cipher-text are links to insecure resources.

For example, a user requesting my tweets will get an object which contains a link to my avatar image.

Twitter is currently returning the insecure link:

"profile_image_url" :
    "http://a2.twimg.com/profile_images/1283757621/Sketch_Avatar.jpg"

Twitter should be returning the secure link:

"profile_image_url" :
    "https://si0.twimg.com/profile_images/1283757621/Sketch_Avatar.jpg"

Exploiting This Weakness

A user (Anna) will request the encrypted text of my tweets
She then requests the unencrypted image.
An eavesdropper (Eve) is listening in on the connection between Anna and Twitter.

Anna ---->Eve---->Twitter  (Secure request)
Anna <----Eve<----Twitter  (Secure response)

When Anna makes the initial request to Twitter, the malicious Eve can’t see what they’re talking about.

• The request “https://example.com/twitter/edent” is itself encrypted. Eve only sees an encrypted request to example.com – not “twitter/edent
• The response containing all the tweets is also encrypted

Anna ---->Eve---->Images  (insecure request)
Anna <----Eve<----Twitter  (insecure response)

Anna then makes the subsequent request for the twitter user’s image, a malicious user can see

• The URI of the request.
• The content of the image.

Impact

Truth is, this has a pretty low security impact.

• There is no way to determine a user’s name based on the URI for their image. (Unless you already have both).
• An eavesdropper has no way of knowing if the image is from the timeline, a reply, a DM, a search, a retweet, or the public timeline.
• Images may be locally cached by the user’s browser – so frequency analysis isn’t reliable.
• A malicious user could alter the image in transit.

Worst case scenario is that if a malicious man-in-the-middle knows which images relate to which Twitter users, they know the intercepted user has seen at least one tweet from that user.

Let’s say Anna is communicating with Bob. Eve is trying to eavesdrop.
If Bob has never tweeted, and Eve sees repeated requests from Anna for Bob’s avatar, she may reasonably surmise that they are exchanging DMs.

Overall

This is a pretty low-impact privacy risk.
It can be fixed by Twitter’s API returning HTTPS URIs where possible.
In the meantime, developers can replace “http://a2.twimg.com/” with “https://si0.twimg.com”.

Dabr’s strength for developers is two-fold

1. Dead easy to install. Unzip the files, fill in your API key(s), upload, done.
2. It’s under an incredibly permissive Open Source MIT License. Essentially anyone can do anything with the code and they don’t need to ask permission, nor contribute anything back to the code base.

It’s a really popular tool. Although it’s hard to count how many users are on the main Dabr server – not to mention all the clones – it’s obvious that many people find it an indispensable way to access Twitter.

According to the New York Times, Rafinha Bastos is the most influential person on Twitter. Guess what client he uses?


So, with a kick-arse product and no silly “Intellectual Property” restrictions, we have let a thousand flowers bloom. There are clients deployed all over the world. Including on my secure server – SSL Dabr. And here’s one more…

Welcome UberSocial!

The most recent flower is UberSocial. As well as having the most popular BlackBerry Twitter client, and an iPhone client – they’ve now beta-launched a mobile web client – based on Dabr!

I’m really excited to see Dabr spreading far and wide. The intention – for me – has never been to make pots of money from it. The adverts on SSL Dabr just about cover the hosting costs.

What I like seeing is my code being used far and wide – and helping people. I know it’s corny, but seeing my code being used by Twitter users in Egypt during the uprising was overwhelming. In a tiny, almost insignificant way, I helped.

I don’t think my coding is good enough to get something into a really important project – like the Linux Kernel, or the International Space Station – but seeing millions of users is personally edifying.

So, long live Open Source and long live Dabr!

The first time was after I called a particularly nasty company “twunts” over a dispute I’d had with them. I’ll be the first to admit that it wasn’t a particularly mature reaction – but I’m not sure it warranted taking a screenshot of the tweet, threatening to show it to the CEO of the company I worked for, then continually calling the company to complain about me. I was a private citizen, not tweeting on behalf of his employer. Luckily, my employers were very good about it and supported me. There is no small measure of schadenfreude when I read about my blackmailers current protracted legal difficulties.

The second time was rather different – and in many ways nastier. I was at a private function and twitpic’d a photo of a (very) minor celebrity. I was immediately contacted by his PR team saying “please don’t tweet anything he says – it’s a private event.” Not a problem for me, I emailed back saying I’d keep schtum.
A few weeks later, I got hauled over the coals and threatened by his “team” for “invading his privacy” and “endangering him by revealing his location.” All over a blurry twitpic which, if they’d asked at the time, I would gladly have removed. Again, threats were made to me about the consequences of my tweets.

Both tweets have now been deleted.

All of which has left me with a rather sour outlook. I now think twice before I tweet anything. Who knows what innocent, flippant, or satirical content can be taken, twisted out of context and then used against you?

#welovebaskers

All of which brings me, in a round about way, to a cause célèbre on twitter – @Baskers. I’ve met her a couple of times at social occasions, but I wouldn’t say I knew her well. Even though enough has been written about her to fill a book, but I’d like to add my tuppenceworth.

It would be the height of egocentricity to claim my experiences were anything like hers. I’ve neither been pilloried in the press nor caused my employers publicly defend me.

But there is a similarity. We are both victims of bullying. It took me a little while to realise this but, since doing so, it has made the world a lot clearer.

@Baskers was the victim of a vicious and petty bullying campaign in two national newspapers.

Perhaps she got picked on because she’s a smart and successful woman. Bullies hate those more successful than themselves.
Was it because she is witty, intelligent, and able to eloquently express her opinion that lead her to be despised by talentless hacks?
Or, as is so often, was she a victim of someone who can only make themselves feel good about their pathetic and wretched life by randomly lashing out at a weaker target?

Whatever the reasons, it’s abuse. Using a position of power to bully someone who can’t respond on the same scale is disgusting – especially from papers which claim to be against bullying.

What Should We Do About Bullying?

I was told by countless teachers that you should ignore bullies and eventually they go away. That may well be true for some childhood scrapes – but removing oneself from the vengeful eye of the gutter press isn’t just as simple as closing the curtains and hoping they go away – as Zoe Margolis found out.

So, can we go down the “official” routes to complain? Sadly, the game is rigged.

In the case of the UK’s legal system – the costs are prohibitive, the players all work against you, the rules are obscure, and even if you win – you’re still tainted by association and may not recover your costs.

In the case of the the Press Complaints Commission the game is so one sided that you stand virtually no chance of success.

• The body you complain to is populated with senior members of the UK’s newspaper industry – so very little chance of a fair or impartial hearing.
• The PCC limit the scope of the complaints you can make about their members.
• If a newspaper doesn’t like the way the PCC timidly pursues it – they can simply leave the PCC and have all cases against them dropped.
• The chair of the PCC, on the basis of this BBC interview, is unrepentant in her stewardship of such an obviously incompetent authority.

Unsurprisingly, the PCC once again looked after their members’ interests.

Revenge?

But what can we do? How do we stop these hideous cyber-bullies?

It’s tempting to think of revenge. It’s a basic human instinct. I’m salivating at the thought of extracting retribution – both on my behalf and by proxy for @Baskers. I’d love to set a pack of ravenous paparazzi on the editors of those newspapers and hound them until they couldn’t take it any more. I’d love to fill the pages of other newspapers with a long exposé of their reports weird sexual habits (as an aside, subscribe to Private Eye and read Street of Shame – they’re the only paper which reports on other papers).
I want to set up website detailing every miss-step a journalist has ever made – and SEO it so that when potential employers search for them, all they find is the very worst.
I want parody twitter accounts dedicated to making them a laughing stock.

I’d love to watch newspaper editors and journalists in utter despair as their children come home weeping because of what the other kids are saying about their mummy or daddy.

And that’s why revenge is wrong. It’s morally wrong to stoop to their level. Hurting other people just traps you in a cycle of violence. Revenge is usually indiscriminate and hurts far more than the target.
Besides, as they say, “never fight with a pig. You both end up covered in shit – but the pig likes it.”

I’m not saying we should passively resist them. We can’t let them beat up our friends and hope they get tired and go away. We can’t simply watch bullying ruin life after life. We can’t close our eyes and wish for the best.

What Can We Do?

I’m a hippy. I think it’s up to us to take a stand in what we believe in. To be the better person. Ultimately, to forgive and show the bullies that there is a better way.

I’m inspired by Michael Lapsley – my wife’s uncle – and the work of The Forgiveness Project. Michael, and the others on the site, have managed to find forgiveness for those who have done them dreadful wrongs.
I can’t even begin to compare Michael’s horrific and permanent injuries to the next-day’s-chip-paper assaults on members of our community. Although I do know the mental scarring from bullying can be as painful as anything which can be done to our bodies.

Reading through the stories on the site one thing becomes clear. It’s important for both parties to understand each other. To understand where the pain comes from and why it was dished out so callously.

I don’t know how we make this happen. I don’t know whether it’s as simple as inviting Quentin Letts out for a drink. Chatting with him and letting him see what pain he has wrought. Is it a case of “adopting” a journalist and showing them how twitter works – and helping them make friends?

How do we reach out to these bullies? How do we show them that they don’t need to fear us? How do we make them understand that their words have consequences? How can we make them see that picking on people isn’t big, and it isn’t clever?

I don’t want (some) newspapers and (some) blogs trying to ruin the lives of those who have caused them no harm. Is that really an impossible task?

Long time readers will know that I have some severe usability and security concerns with Twitter’s OAuth implementation. See also my interview in The Register.

Zach Holman has an entertaining and informative blog post about giving Twitter applications fine grained controls.

Essentially, he’s saying that you should be able to authorise an app for just posting, for example.
Here’s his graphic which I’ve stolen.

This doesn’t go far enough.

I was taking a look at this LinkedIn application which graphs your contacts.

Take a look at their OAuth screen.

At the bottom is an “Access Duration” option – giving you the option to try out the app and have it automatically revoke after a specified period of time.

Now, this isn’t something you’d want to do for every app. But it gives you a method to limit the damage that a malicious app can do. Remember, just because an app isn’t malicious today, doesn’t give you any guarantee about its future performance.

As it happens, the Oauth Specification 2.0 has this to say in section 4.2.2. Access Token Response

expires_in
         OPTIONAL.  The duration in seconds of the access token
         lifetime.  For example, the value "3600" denotes that the
         access token will expire in one hour from the time the response
         was generated.

If you run a service relying on OAuth, please consider giving users an Access Duration option.

Here’s some basic code for an Android app which will post the URL of your app to Twitter. Stick it in a button or menu item for easy sharing.

Some important things to note.

1. This is set to post to the mobile version of Twitter.  Your user is on a phone – don’t direct them to a site that won’t work on their device.
2. The second string is URI encoded.
3. Consider if you want to post a “market://” link.  I would advise against it.  Twitter won’t render it as a link and, even if it did, 90% of users won’t be able to click on it.  Make it a link that will direct desktop users to your website, mobile users to a mobile friendly site and Android users direct to the market.

Facebook also has an API for this sort of sharing.

http://m.facebook.com/sharer.php?u=example.com&t=test

Again, it points to the mobile site and needs to be URL encoded.

Happy sharing!

Pagination is the act of splitting data into logical  pages. Suppose I had a list of item, numbered 0 – 99.  If I want 20 items per page, it’s trivial to see that pagination looks like:

p1 = 0-19
p2 = 20-40
p3 = 41-61
p4 = 62-82
p5 = 83-99

If I wanted to start at, say, page 55 – pagination would look like:

p1 = 55-75
p2 = 76-96
p3 = 97-99

Easy, right?  So why am I telling you this?

Twitter Timeline

Imagine that those items are Twitter Status ID.  Each one represents a tweet in your timeline.

Twitter will allow us to “page” back and forth through our timeline. If we say status ID 80 is the most recent post in our timeline, and we want to see 20 tweets at a time, pagination would look like this.

p1 = 80-60
p2 = 60-40
... etc.

Normally, that would be fine.

The only issue is that friends are posting all the time.  Imagine we start with tweets 80-60.  We go to page 2, but in the meantime, 5 new tweets have been made.

p1 = 80-60
p2 = 65-45

The user sees 5 tweets she has already read.  Not desirable.

If 20 tweets had been made before clicking on the “next” button, this is what happens.

p1 = 80-60
p2 = 80-60

Max_ID To The Rescue (AKA, the easy bit)

Luckily, Twitter allows us to use a max_id parameter in our API calls.  This says “Get the tweets older than this number.”

http://api.twitter.com/1/statuses/home_timeline.json?max_id=123465789

So, using max_id we can ensure that the user never has to read the same tweet twice.  Instead of dumbly using pages, we call the specific tweets we want.

p1 max_id=80 = 80-60

p2 max_id=60 = 60-40

Easy!  We just use the oldest tweet on the page as the max_id parameter when we call the next page.

Looking To The Future (AKA, where it all goes horribly wrong)

So far, we’ve looked at stepping back in time.  Seeing older tweets.  Suppose we want to see newer tweets?

Twitter provides us with a since_id parameter for API calls.  This says “Get the tweets newer than this number.”

Unfortunately, it doesn’t work.  Well, it works but not the way I expected it to!

Suppose our user is deep down in her tweets, this is how I would expect since_id to work

max_id=60  = 60-40

 (So, let's show any more recent tweets)

since_id=60 = 80-60

We see the 20 tweets that occured since the since_id.  Right?  Wrong!  This is what happens?

max_id=60  = 60-40

(So, let's show any more recent tweets)

since_id=60 = 100-80

What?

An Explanation

The since_id retrieves tweets starting with the most recent.  It stops when it reaches the since_id.

I don’t know the max_id that I’m looking for, so I can’t call that.

I could call the most recent 200 tweets and look for the 20 I need.  That’s wasteful in terms of bandwidth and processing – there’s also no guarantee that the since_id will be in there.

So, I have a problem.  The “Older” link in my Twitter application will work.  The “Newer” links won’t.

Any suggestions?

You can see the full code on Dabr’s Google Code page.

First of all, you’ll need to have enabled OAuth for your Twitter client. I use Abraham’s excellent OAuth libraries for PHP.

This tutorial assumes you already have OAuth working. I’ll attempt to explain what I’m doing as I go along – but the code should be pretty readable.

Start by reading the Twitpic API documentation. You will also need to register for an API key – this only takes a few seconds.

We start by setting CURL’s headers to those required by Twitpic

Next, we create the OAuth credentials that we need. Essentially, we’re signing a URL request. We then pass that on to Twitpic and they verify it with Twitter. We never pass our OAUTH_CONSUMER_SECRET – so Twitpic can’t impersonate us.

We add these generated credentials into the header.

Add everything into CURL

The data we send to Twitpic (message text, image and key) have to go via POST.

All done, we now send the data to Twitpic.

If this has worked, Twitpic will pass back the URL of the image we posted. We then need to post the entire message to Twitter ourselves (Twitpic can’t do it for us).

This next part is specific to Dabr – your client may post things differently.

If it didn’t work, Twitpic will tell us why.

The easy bit.

It’s easy to post the data to Twitpic

$media_data = array(
	'media' => '@'.$_FILES['media']['tmp_name'],
	'message' => html_entity_decode($_POST['message']),
	'key'=>'123465789132465'
);
curl_setopt($ch,CURLOPT_POSTFIELDS,$media_data);

OAuth Credentials

Using Abrahams OAuth library for PHP, it’s easy to get the required OAuth data.

require_once('OAuth.php');
// instantiating OAuth customer
$consumer = new OAuthConsumer(OAUTH_CONSUMER_KEY, OAUTH_CONSUMER_SECRET);
// instantiating signer
$sha1_method = new OAuthSignatureMethod_HMAC_SHA1();
// user's token
list($oauth_token, $oauth_token_secret) = explode('|', $GLOBALS['user']['password']);
$token = new OAuthConsumer($oauth_token, $oauth_token_secret);

// signing URL
$fakeurl = 'https://twitter.com/account/verify_credentials.xml';
$request = OAuthRequest::from_consumer_and_token($consumer, $token, 'GET', $fakeurl, array());
$request->sign_request($sha1_method, $consumer, $token);
$OAuthurl = $request->to_url();

The Tricky Bit

I’m following the header example in the API documentation. Passing these variable to Twitpic is where I seem to go wrong.

$header = array(
'X-Auth-Service-Provider: https://api.twitter.com/1/account/verify_credentials.json',
'X-Verify-Credentials-Authorization: OAuth realm="http://api.twitter.com/"'
);

I then modify the second header so it reads

"X-Verify-Credentials-Authorization: OAuth realm="http://api.twitter.com/",
oauth_consumer_key="aaaaaaa",
oauth_nonce="bbbbbbbbbbb",
oauth_signature="ccccccccccccc%3D",
oauth_signature_method="HMAC-SHA1",
oauth_timestamp="123456798",
oauth_token="15948715-dddddddddd",
oauth_version="1.0""

The Error

401 “Could not authenticate you (header rejected by twitter).”

GAH!

One of the advantages of the text processing is that it will recognise that www.example.com is a URL and automatically create a hyperlink. Considering that dropping the “http://” represents 5% saving on Twitter’s 140 character limit for messages, this is great.

So, I was mightily surprised to get this bug report from user “schmmuck”

Dabr rendering error

How very odd… This is how it looks on m.twitter.com.

m.twitter rendering error

Twitter also use mobile.twitter.com for smartphones. Here’s how that site renders the text.

mobile.twitter rendering error

Finally, let’s take a look at the “canonical” rendering at Twitter.com

Twitter rendering error

The Problem(s)

The first issue is inconsistency.  Twitter ought to be using the same regex for each of its sites.  It doesn’t.  This means that different developers will get divergent experiences.  This leads to confusion, which leads to fear, which, as we all know, leads to anger…. and so forth.

Secondly, and more importantly, parsing is hard.  There are so many edge cases that errors inevitably creep in.  My post about hashtags explains the problems in defining what should be recognised.

So, based on what we’ve seen, should Twitter recognise any of the following as URLs?

news.bbc.co.uk – no www there.

invalid.name – a silly URL, but a valid one.

खोज.com – International domains contain more than just ASCII

All the above are valid – yet they’re not recognised by Twitter.

A (Simple) Solution?

There is a canonical list of TLDs which is also available as a plain text list.

Any string containing a “.” followed by a valid TLD, then followed by a space or “/” should be treated as a URL.

Your thoughts?

@@ and geotag

Reply to All

The @@ symbol allows you to reply to all the people mentioned within the tweet.
It only shows up on tweets which mention other users – so you should only ever see it when it can be used.

Hitting @@ on the above tweet will pre-populate the text box with “@topgold @whatleydude @dabr”.

It should remove any duplicates and should also remove your username.

Geotagging

Twitter recently allowed the geotagging of tweets. This allows you to set the longitude and latitude of where you are when you wrote your message.
Clicking on the green globe (it just looks like a dot at that size!) takes you to a mobile mapping service.

Google Maps Mobile

It only shows up on tweets which are geotagged – so you should only ever see it when it can be used.

How Does It Work?

You can see all the code in the SVN, but here’s a quick rundown.

Geotagging is very simple.  Statuses now have a <geo> field.  If this is populated, it will contain a longitude and latitude.

We can take those values and pass them to Google Maps Mobile (or your favourite mobile mapping service).
if ($geo !== null)
{
   $latlong = $geo->coordinates;
   $lat = $latlong[0];
   $long = $latlong[1];
   $actions[] = theme('action_icon', "http://maps.google.co.uk/m?q={$lat},{$long}", 'images/map.png', 'MAP');
}

Reply to all is a little more involved.
Twitter provide an excellent set of Text Libraries. These provide officially sanctioned regular expressions to help your project find URLs, hashtags and usernames.

Using the Twitter Text Extractor class we can determine how many usernames are mentioned in a tweet. Any duplicates are removed – as is the user’s username.

Any questions – just ask!

I wracked my brains thinking about how they could have gotten in there before I realised it was a long-dormant friend who had changed their name and avatar.

But, in thinking about how a spammer could infiltrate one’s timeline, I think I came up with a fairly bullet-proof method to spam Twitter users.

I present this as an exercise in devious thinking – and also to show how our assumptions about security can play against us. Remember, hacking and impersonation are likely to be illegal in your jurisdiction.  This information is designed to help you understand how security weaknesses can occur.

Being Evil

Imagine you are a nasty, evil Twitter spammer. Your own mother wouldn’t spit on you if you were on fire – that’s how mean you are. Here’s what you do.

1. Obtain a user’s password.  Admittedly, this is the hardest part of the process. You might use a dictionary attack, use the same password they use to log in to another site, or somehow steal it.
2. Log on to Twitter.
3. Go to “Connections” and see which services they have connected to using OAuth.  For the purposes of this experiment, let’s assume they use Example.com.
4. Go to Example.com and OAuth yourself with Twitter using your mark’s credentials.
5. Here’s where the ordinary spammer falls down.  The ordinary spammer will start sending out messages from the mark’s account.  That’s not the aim of this weakness.
6. From the mark’s account, through Example.com, make your victim follow one of your spam accounts.  An account which exists solely to show adverts to your victim.

Your victim now sees your adverts for pills, poker and porn in their timeline.  With any luck, they’ll just assume that one of their true friends is promoting your illicit wares.

Counter Attack

Most victims will assume that they accidentally followed your spam account – or that one of their friends has been hacked.

Worst case scenario, they unfollow your spam account.

So you just make them follow you again! Remember, you are still OAuth’d to Example.com. You can make them follow as many of your spam accounts as you think you can get away with.

At this point, the intelligent victim will think that their account may be compromised and change their password.

It doesn’t matter! Because you have used OAuth, password changes don’t affect you.  You can continue make them follow as many of your spam accounts as you think you can get away with.

At this point, the really intelligent victim will go through their OAuth connections to look for something suspicious.  They won’t find it.  Remember steps 3 and 4?  You are OAuth’d to a service that your victim trusts.

Because of the way Twitter displays OAuth information, there’s no way for a victim to know when a service was last authorised.

Twitter OAuth Connections

There is no information other than the first time the OAuth was set up.  No last accessed date, no IP addresses, nothing useful.

When following an account, the victim gets no notification of what has happened, when it has happened or how it has happened.  There is no way of them knowing which of their OAuth’d connections have been compromised, nor when it happened.

Their only safe option is to revoke every single OAuth connection.  Then reauthorise.  A time consuming and annoying prospect.

Conclusion

I hope I’ve demonstrated two things.

Firstly, there’s more to spam then just sending out messages.  Forcing someone to read a message is just as annoying.

Secondly, our understanding of security and usability haven’t quite caught up with the new tools which are available to us.  OAuth is still better than giving your password to an untrusted site – but without essential usability changes, a compromised account is a lot more dangerous than the user would suspect.

This “attack” still relies on a victim having their original password compromised.  That’s not a trivial matter.  But security is like sexual health – it only takes one little accident…

#hashtag – As long has there has been a way to search Tweets* people have been adding information to make the easy to find. The #hashtag syntax has become the standard for attaching a succinct tag to Tweets.

The Twitter Engineering Blog

That’s all well and good, but as I discovered yesterday, without standardisation the ability to search falls apart.

I’m not talking about whether you should use the #LondonFire tag rather than #FireOfLondon or #LDNfire. Rather; how does a computer recognise what a valid tag is?

Why Does This Matter?

Search and tracking quickly break down if they are inconsistent.
For example, if you are using #Romeo&Juliet to mark all your conversations about the play you are watching, different Twitter clients will link through to either #Romeo, #Romeo&, or #Romeo&Juliet.  Each search returning potentially different conversations.

What’s The Convention?

Twitter’s website ought to be the definitive source of how hashtags work. This is their main site.

Twitter Website Hashtag

Yet, when we visit their mobile site – we get a completely different experience.

Mobile.Twitter's hashtags

Application Confusion

Because there aren’t any widely publicised definitions for what hashtags are, some applications have a significantly different attitude to hashtags

SocialScope Hashtags

UberTwitter's Hashtag Support

Standardisation

To be fair, the Twitter team do have a standard.  Even if they don’t use it themselves.

They even have some limited test cases and libraries in Ruby and Java.

So, given that Twitter, their implementation and apps all disagree on what a hashtag is, let’s try to work our what they should be.

Anatomy of a Tag

To begin at the beginning.  A hashtag starts with a hash. #.  Simple, no? No.

There are two different hash symbols! There’s the # we all know and love, and there’s ＃.  Looks pretty similar, but in fact it’s the unicode symbol [U+FF03]

Actually, that’s not the beginning.  What comes before the # of the hashtag?

Consider the following examples – which should be hashtags?

• #tag – the # starts off the Tweet
• This is my tweet #test – the # comes after a space.
• This is it.#tag – the # is pushed against some punctuation, perhaps for reasons of space.
• Here we go-#LiftOff – the # is pushed against a -
• I’ve run out of space#OhNo – the # is pushed against some text
• &#nbsp; – the # is part of an HTML entity
• text　#hashtag – the # comes after a “wide space” (U+3000)
• Should I use #tag/#hashtag? The # comes after a /
• Is this valid ##tag – there are two #s

So, we can see it’s a little more complicated than we first thought.

The End

Let’s skip over what’s in a hashtag and as “how do we know that a tag has finished?”

Consider the following examples -

• New album #OMG! – should the ! be part of the hashtag?
• #BreakingNews: dog bites man – should the : be part of the hashtag?
• (is this a #tag) – should the ) be part of the hashtag?
• I like #tags#

We probably don’t want to have any punctuation at the end of our tag.  Can you think of any counter examples?

Yummy Filling

Our language is more than just the letters A-Z. We’ve got punctuation, numbers, symbols and all manner of other glyphs.  Which of them count as part of a hashtag?

Take a look at these examples

• Vote Bush! #Don’t
• My dog died #:-(
• Einstein #e=mc^2
• I’m on bus #123
• I’m giving #110%

Using Twitter’s standards, none of the above render as complete tags.

Foreign Languages

We’ve mentioned accents above.  As we can see in the first example, “funny” characters can cause problems.  Broadly speaking, there are three issues.

1. Accents.  Should the é on #Café be linked?
2. Accents.  Is #Romeo the same as #Ŕöméø?
3. Japanese, and some other languages, don’t use spaces.  Is #tagの valid? What about # 会議中 ?

Exhausted

These are a fraction of the possible problems.  It’s exhausting trying to find all the possible textual combinations and permutations which could lead into a hashtag.  No wonder there is confusion!

Search is a complex, profitable, and useful business.  It’s of vital importance that there is a legitimate, comprehensive standard which all sites and applications can follow.

This sort of knowledge is common in life – but is fatal in computing and design. Take the following tweet I received.

Mawkins

(Mark Hawkins)

@edent @dabr you folks aware ampersands / &s don't seem to work as part of hashtag links?

On 24-2-2010 16:27:19 from TweetDeck in reply to Terence Eden

The complaint was that #tfm&a should be rendered as #tfm&a not #tfm&a.

Everyone knows that’s how hashtags work!

On Twitter’s website, find the page which discusses hashtag syntax. Find where they explain how they should be styled.

You can’t.

And thus implicit knowledge is born. Dabr only looks at letters and numbers in a hashtag. It assumes that any other character is the end of the tag.

Dabr's Hashtag

Without official guidance – implicit knowledge develops.

Has Dabr Got It Wrong?

No. I don’t think so. Take a look at how Twitter on the web renders hashtags…

Twitter's Web Site

…and on the mobile.

Twitter Mobile

So Where Does Render The Full Tag?

Several applications don’t render tags in the same way as Twitter. Take a look at SocialScope

SocialScope Hashtags

Tweetie2

I’ll upload more screenshots if I find examples of “badly behaved” hashtags.  Please let me know if you find any.

What Does Twitter Say?

Twitter has one page devoted to hashtags.  It is a support page for hashtags.  This explains to people what hashtags are.  There’s no detail on valid characters, maximum length, or any of the things which might be useful for a developer or designer.

Edit 2010-02-25

David Dorward has pointed out that there is an official resource. On the Twitter Engineering blog – which isn’t linked to from the developer site – there is a page discussing hashtags and how to validate them. You’ll notice that they are rather circumspect on what should constitute a hashtag.

Conclusion

Standards and guidelines allow developers to create compatible applications.

Without explicit recommendations, developers will diverge as widely as possible.  Twitter – and everyone with an interest in compatibility and usability – needs to ensure that the knowledge they impart is explicit.

Letting people make it up as they go along leads to confusion.

It’s one of my gripes with Linux.  If you’re editing a configuration file, you are relying on yourself to sanity check your input – often without knowing what the limits are.

Take these two different examples.

In a text file, we might have:

#Maximum Widgets to fidget
maxW_to-F = 0

Whereas a GUI would show

How many Widgets do you want to fidget?
1235

Even if you don’t know the rules behind Widget fidgetting (must be a prime number lower than 7), the GUI won’t let you choose a value that you can’t select. The GUI doesn’t prevent you setting an innapropriate value – just an illegal one. Your config file, however, could be set to any crazy value that a user type – often resulting in “unpredictable” results.

#Maximum Widgets to fidget
maxW_to-F = seventeen

It’s with this in mind that I’ve made the following change to Dabr – the mobile Twitter client.

To Auth or Not To Auth? That Is The Question

Twitter’s API has bug / peculiarity (reported to their discussion board) which causes Dabr to log a user out. Let me explain the steps

• User 1 (@private) has set her tweets to “protected”.
• This means no one can see @private’s tweet unless she allows them.
• @private has not allowed User 2 (@edent) to view her tweets.  She is protect from his view.
• @edent clicks to view @private’s profile.
• @edent can see that @private has 42 friends, 17 followers and 3 favourites.

So far, this is the same behaviour on Twitter’s website as it is through their API.  Here’s the difference

Web

• @edent tries to see @private’s followers and can see their names, profile pictures etc.
• @edent can also see @private’s friends
• @edent cannot see @private’s favourites (or even how many favourites she has)

API

• @edent tries to see @private’s followers, friends or favourites
• Because @edent isn’t allowed to see @private’s info, the API returns 401 Authorisation Required.

This is where things get tricky. Dabr sees the 401 and concludes that the user has invalid credentials.  It then, as a security measure, clears the user’s cookie and logs them out.

This may be a little harsh, but HTTP 401 essentially says that the authorisation has failed.

Fixing It

There are 3 ways that this could be fixed

1. Twitter could rationalise the API to allow access to the same content that a web user gets.
2. Twitter could return a different status code.
3. Dabr needs fixing.

So, how do we get Dabr not to log out when it receives a 401 only under these specific circumstances?

Looking at the code, we can see that Dabr simply sees the HTTP response code.  To fix it we’ll need to pass extra parameters, check where the code was called from, investigate all the edge cases, add more logic to the system, futz around breaking things, etc… etc…

What a pain in the…

Or

Don’t let users do things they can’t do.

If a user can’t see the information – why do we even let them try to see the information?  Why can’t we just get rid of the link?

This is what a user currently sees:

Old Style

We’ve established that they can’t view followers, friends and favourites.  So we can get rid of those links (but not the information).

New Style

(Incidentally, I’ve changed the order of the links.  I’ve tried to group together similar items.  Followers, friends, favourites and lists go together. Then DM. Finally, follow, block, report spam.)

Now a user cannot click through to an unwanted error message.

Or

There is another way round this.  With “Direct Messages” we could do the same thing – simply remove the link if you’re not able to send that user a DM.

Instead, we’ve taken the approach of displaying a suitable error message.

Direct Message Warning

The advantage of this is that the user gets an explanation as to why they are unable to complete an action.

Your Thought?

Which do you prefer? Being unable to click on a link (with no explanation) or clicking on a link only to be given a warning message?

NB1: I primarily use SocialScope on my BlackBerry. SocialScope’s terms of use prohibit me from showing screenshots of their beta. Hey, guys, I want to show everyone how great you are!

NB2: Screenshots taken on a BlackBerry 9000 running OS 5.0

First Impressions

I’ve never understood the need for a EULA. They’re long, confusing, boring and a terrible way to make a first impression on your customers.  Twitter for Blackberry has a 30 page monstrosity.

EULA Hell

No one reads them, no one cares, find a better way.

Good

This app really shows off the power of the BlackBerry SDK. It’s a pity that more developers don’t make full use of it.

Tweets show up in your message list….

Message Integration

…and can be opened from there.

Open From Message

Images can be sent directly to TwitPic.

Send To Twitter

Apps like SocialScope and UberTwitter also hook in to the OS so that you can click on any #hastag and @name and be taken directly to that page within the app. I wasn’t able to ascertain whether this functionality was available in this app.

Bad

Limited view space.  At best, you can view 2 tweets at a time before scrolling. That’s not a very efficient use of space.

The Tweets Went In Two By Two

Poor navigation.  You can’t simply scroll left or right to move between pages, you need to scroll all the way to the top, or open the menu.

Navigation

The software also doesn’t remember which tweet you were on before you left.  It always refreshes the timeline and jumps to the top.  This is really annoying and it’s a problem solved in both UberTwitter and SocialScope.

Ugly

Missing loads of features that many users have become accostomed to.

• URL expansion.  bit.ly/fjhgkdfhg means nothing – use their API to expand the URL
• Embed images and media. Show a preview of flickr images, YouTube videos etc.  At the moment, it only appears to do TwitPic.
• No OAuth!  This is partly due to Twitter’s broken mobile OAuth support.
• No autocomplete. SocialScope will show a list of your friends when you type @ to allow you to quickly select them.

Conclusion

This app is amazing… if you’ve never used UberTwitter or SocialScope.  I also found it to be marginally slower than the other apps.

This was a problem I faced when trying to get Dabr to render the ASCII* art produced by Aral Balkan‘s Feathers App. Feathers uses line breaks to achieve images like…

〰❀❃ Introducing ❃❀〰
My new iPhone app,
░░░▒█ Feathers █▒░░░

What’s confusing to me, as a developer is the inconsistent way Twitter handles line breaks. For example…

Twitter’s website ignores line breaks.

Desktop Spacing

Twitter Mobile, however, does line breaking correctly.

Android Spacing

Twitter has no style guide for rendering and – if it did – appears to render inconsistently. So, what’s the correct thing to do?

In my opinion, respect the user. If a user has indicted that they want something, it is a developer’s role to implement it (unless it causes severe problems for the system).

How to achieve it?

You could use all sorts of complex regular expressions to get line breaks and convert them.
Thankfully, PHP has this function built in with nl2br() which will give you (X)HTML line breaks wherever a break occurs in the text.

Footnote

*Technically, Unicode Art. But that doesn’t have the same ring to it.

Usually, I would be a big fan of this move – especially if it forces password anti-pattern sites like TwitPic to implement the new, secure standard.

This means that you won’t be able to log in to a third party site by giving them your username and  password.  You will have to use OAuth to securely validate with the main Twitter site.

But – as ever – there’s a dark side to OAuth.

Repressive Regimes

One of the joys of Twitter is that its clients are decentralised from the main site.

This means that if Twitter is blocked in your country, you can use a third party client (like Dabr) to access it.

Twitter User -> Dabr -> Twitter API -> Dabr -> User

If Dabr became sufficiently popular to be blocked by an oppressive regime, you can switch to any one of the thousands of Twitter web clients.

OAuth forces you to the main Twitter site.  While you may visit Dabr to start with, you would be redirected to Twitter to complete OAuth.  If Twitter is blocked, you can’t connect.

At a stroke, Twitter has shut itself off to anyone in a repressive country.

This has been picked up by some concerned users.

A (Hacky) Way Around It

There’s really only one way around this problem.  The third party web client has to act as a man-in-the-middle.  There’s a patch for Dabr – developed by cnyegle – which will ask for a username and password, then proxy it to Twitter, get the OAuth token and tweet on behalf of the user.

From the user’s point of view, they are still giving the (potentially untrusted) site the username and password.

Challenge Response

This could be solved by implemented a challenge / response system.

1. Alice visits the Dabr website.
2. Dabr asks Alice for her username (and only her username)
3. Dabr asks Twitter for the challenge associated with Alice’s username.
4. Twitter checks that Dabr is an authorised website (i.e. has signed up for OAuth).
5. Twitter returns the response:  A secret phrase which Alice has previously chosen.
6. Dabr displays this phrase to Alice.
7. Alice knows that Twitter trusts Dabr
8. Dabr asks Twitter for the password challenge.
9. Twitter returns that it requires the 3rd, 5th and last character from Alice’s password (the characters requested change randomly).
10. Dabr asks Alice for only those characters.
11. If Alice provides the correct characters, an OAuth token is granted to Dabr to tweet on behalf of Alice.

This has the advantage of proving that Dabr is trusted (by displaying Alice’s pre-defined secret phrase) and mitigating the risk that Dabr is untrusted (by only revealing part of the password).

Conclusion

This is a very new area, and I’ve not had a chance to read through all of the proposals.  Nevertheless, it remains a fundamental problem that, if you can’t access a site, you need to delegate your trust to someone else.

I’m not a security expert – so I would appreciate someone pointing out the flaws in my reasoning.

For a service which started out operating by SMS, Twitter takes a surprisingly unenlightened view of mobile. It’s main mobile service – http://m.twitter.com/ – is almost completely devoid of useful features. That’s one of the main impetuses behind the development of Dabr. Their latest mobile site – http://mobile.twitter.com/ – is really only suitable for the tiny minority of people who have smartphones.

So, understandably, many people use 3rd party sites like Dabr. They are now faced with a dilemma – give an untrusted site their username and password or try to use OAuth on the mobile.

A few weeks ago came the announcement that OAuth was finally ready for mobile… Was it? No. Once again a “mobile friendly” site designed with masses of JavaScript and guaranteed not to work with the majority of phones on the market.

Here’s how mobile OAuth looks on a variety of popular mobile phones.

BlackBerry

BlackBerry Twitter OAuth

While this looks pretty enough, it doesn’t work. The buttons aren’t clickable. I’ve tried with and without JavaScript. No matter where I click, nothing happens.

Android

The Android’s User-Agent isn’t detected by Twitter as being a mobile phone. While it’s true that the browser is very capable – the OAuth screen is a lot more usable when it’s in mobile mode.

Android Twitter OAuth

So, it works, but it doesn’t look nice.

N95

The N95 makes a good test phone because it’s popular. Probably more popular than the iPhone.

N95 Twitter OAuth

N95 Twitter OAuth

It’s not pretty – but at least it works.

Others

The Sharp GX-10 is my default test phone. One of the first phones with a colour HTML browser. If your site can work on this phone, it will work on any phone. There are no screenshot capabilities for this phone – but rest assured, it does not work.

The three phones I’ve demo’d above are very popular modern phones – AKA the minority. If they don’t work well, what chance for the people using older phones?

Useless! How hard can it be? All it needs is a username field, a password field and a button. That’s just about the most basic page imaginable. It should be child’s play to make it work on mobile.

This was first raised in March 2009 on Twitter’s issues list. It’s currently the most popular bug.

So, we’re stuck in a dire situation. Third-Party mobile sites get access to Twitter users’ passwords because Twitter are unable or unwilling to develop a simple OAuth form. It would be fascinating to know how many of Twitter’s security breaches are down to corrupt or insecure 3rd party sites which leak passwords.

I’ve tested this to work on the Vodafone 360 H1 – but it should work with any JIL handset.

Because it isn’t certified, you may need to dial

*#35767#

to remove the H1′s security check.

The code is very simple.  The JIL SDK allows you to call specific phone application – in this case, all I’ve done is invoked the browser.

<body>
   <script type="text/javascript">
      Widget.openURL("http://m.dabr.co.uk/");
   </script>
   <div>
      Launching Dabr!
   </div>
</body>

One thing to note, if you’re deploying the widgets from your website, ensure you have set the MIME type to “application/widget” – as per the W3C standards – otherwise many devices won’t recognise it as a valid widget.

If you’re interest in developing for this platform, there’s a €1,000,000 bounty up for grabs.  There’s also a Dev Camp at Mobile World Congress this year.

I work for Vodafone – this is my personal blog, I don’t to speak for them.  All opinions, claims and mistakes are my own.

If you scan in the above QR code, it will take you to Twitter and pre-populate your status with “@edent”. You can, of course, set it to be any message you like.

There are two key things to remember with QR codes.

1. Keep the information in them short to ensure readability.  The more characters you use, the larger the code and the harder it is to scan.  You can use URL shortners to do this if you have a long message.
2. Make it mobile. QR codes are scanned by mobile phones. The information is looked at on a phone screen.  You should always direct people to the mobile version of your site.  In this case – m.twitter.com

Getting the client was fairly simple, but could be better. Simply visiting http://seesmic.com was enough to bring up a mobile friendly page with download instructions.

However, scrolling down presented this mess.

You simply can’t rely on users to know what make, model or firmware version they have. Use the user-agent string to do as much of the hard work as possible. If a user tries to install an incompatible version, it won’t work and they’ll blame you.

The download itself is a sprightly 172KB and installs very quickly – it also doesn’t require a reboot.

Once you’ve signed in, the main interface is very simple. UberTwitter, for example, presents the user with complex set-up options on the first run. Seesmic gets straight to the action.

Seesmic is entirely menu driven – there are no shortcut keys. UberTwitter allows me to hit R for reply – on Seesmic, I have to delve into the menus. Luckily, they’re short and have obvious names.

Reading a tweet gives the usual option, hashtags and @s are hyperlinked – as are web addresses.

On usability flaw, the cursor is at the top of the screen. A user has to scroll through the “Web” hyperlink before getting to the links within the tweet.

Again, because there are no keyboard shortcuts, everything has to go through the menu.

Searching is problematic.

Pressing enter doesn’t start a search, it inserts a new line.
The search button isn’t the first thing you get to when you scroll down.
While the results are standard, this odd error message kept popping up.

I encountered this error several times.

Writing new tweet has some great usability touches – and some real clunkers.

The closer a user gets to filling the 140 characters, the more red the screen turns.

URLs can be automatically shortened and images can be added. This shows one of the great usability failures of Seesmic.

Rather than use the BlackBerry’s file select utility, it uses its own – and it’s dreadful. Ugly looking, no ability to search, slow, no preview. Overall, a real let-down.

I don’t know why companies often insist on creating their own versions of well established system functions. Especially when they add nothing and remove plenty.

Sending a tweet was continually problematic for me.

Although it eventually relented.

Seesmic also makes great use of notifies, showing you on your home screen how many unread tweets you have.

Overall, Seesmic isn’t a bad client. There are a few rough-around-the-edges bugs and the file selection is atrocious but other than that, it works. Power users like me will miss the shortcut keys of UberTwitter, and UT’s system-wide integration (uploading photos, capturing hashtags in emails etc.).  It lacks other features such as in-line photos, ability to see followers and friends, it also has no way of marking a tweet as a favourite or seeing favourites.

One feature it does have – a first in mobile twitter clients – is the ability to view lists.  You can’t add, edit or create – but you can see the lists you have created or are following.

Conclusion

Seesmic works well enough as a basic twitter client for BlackBerry – but there’s nothing exciting about it.

Since missing him play 10 years ago, I’ve caught up with him on The Now Show and Twitter (where I’m happy to report he’s a user of Dabr – the mobile Twitter client I code for).

So now, on to his gig at The Bloomsbury Theatre.  A few hundred people made it through the driving rain on a cold November night – and they were rewarded with the funniest set I’ve seen since… well… since Boffoonery earlier in the week!

The audience were, as Benn put it, a right bunch of weirdoes. A mixture of Radio 4 fans, prog-rockers, students, comedy buffs and – fairly obviously by the way they wooped during the opening bars of every song – some die-hard groupies.

As much as I love The Now Show, it’s obvious that Mitch is much better when freed from the constraints of the 90-second-topical-song format – his long form work is energetic, funny and surprising caustic.  That said, he skilfully took audience suggestions and composed a short topical song – Nudey Bear – during the interval.

For me, the highlight of the evening was “IKEA” – hearing a hall full of people sing like demented Vikings while the band turned the volume up to 11 is a memory I’ll treasure forever.  You can listen to the track for free at We7.

His current tour runs until the 27th of November – walk, hitch-hike or drive to your nearest venue; Mitch Benn shouldn’t be missed.  I know I’m not going to leave it ten years before I next see him.

I received a rather worrying email from Twitter.  Apparently they thought my password had been compromised and needed to be reset.

Reset Your Twitter Password

After checking to see if it was valid, I went and changed my password.  Any site which relied on a cookie to post to Twitter would have been blocked out. Ha! Gotcha, suckers!

The OAuth Problem

OAuth tokens are not revoked when the master password is changed.

OAuth is a great idea – rather than give your username and password to any random site, you log on to Twitter and tell them that you authorise the refering site.  The site gets an OAuth token and never gets to see your password.  Great! Right? Not really.

Let’s consider the following scenario.

Alice has a Twitter username and password.

Bob runs a Twitter site.

Alice visits Bob’s site.  Alice is security conscious and uses OAuth.

Eve somehow discovers Alice’s password.

Eve also visits Bob’s site and uses OAuth.

Alice gets suspicious about strange activity on her account and changes her password.

Because Bob’s site uses OAuth, it does not require either Alice or Eve to re-enter Alice’s password.

In this scenario, Alice has to visit Twitter’s OAuth Connections page and revoke access to all the sites she has previously connected to.  Alice has no way of knowing when each site was last accessed.  She also doesn’t know which site Eve is using.

Twitter's OAuth Page

The Problem

Changing a password should – in the minds of most people – mean that you need to re-enter your password even if you have previously authenticated yourself.

In this scenario, changing the password does not revoke access to malicious users who have previously used your credentials.

Twitter should revoke all OAuth tokens when a user’s password is changed. It is the only way to ensure that stolen credentials cannot continue to be used after a user has changed their password.

Addendum

As I’ve made clear in the comments – this isn’t a vulnerability within OAuth per se.  It’s a usability issue which has strong security implications.

I spoke to Eran Hammer-Lahav (listed as OAuth’s advisory contact) who said:

If you suspect someone stole your password, you should revoke any tokens you did not personally authorized. But there is no reason to revoke tokens just because you are changing password.

While I appreciate this as the official line from those in the know, it does nothing to prevent a user who uses the same sites as you.  For example, I can see on every tweet that you use Dabr.  Therefore, I can safely OAuth myself as you on Dabr.  You’ll change your password, but you won’t revoke Dabr’s token because you personally authorised it.

Continuing The Conversation

Heise Online provides comentary in German (English version)

El Reg has a feature about Twitter and OAuth.

There’s also an interesting discussion over at Hacker News.

Twitter allows you to mark certain posts as favourites.  You may do this for a number of reasons – because you thought it was funny or interesting, because you want to reply to it later, or as a general bookmark feature

They’re Private, Right?

Wrong. You can see any Twitter user’s favourites – even if they’ve protected their tweets. You can’t see the tweets of protected users.  Here are my favourites http://twitter.com/edent/favorites

You can also see who has marked your tweets as their favourites.

What Do Your Twitter Favourites Say About You?

Let’s take a “random” example – The Royal Mail.  They only have one favourite.

njsykora

(Francis Lambourne)

It is written into the CWU constitution that Royal Mail must go on strike once every 4 months. Whiny fucks.

On 14-7-2009 14:15:53 from Echofon

Nice.

If you’re advising on social media policy – be sure to tell your clients that what they mark as a favourite are open for all to see.  What they bookmark may not create a favourable impression.

Now, there may be perfectly legitimate reasons for clicking “favourite” on an obnoxious or offensive tweet. You may want to mark it so you can show your lawyers, blog about nasty people on the Internet or even a simple mis-click with the mouse.  But in the highly fraught world of social media – it pays to think twice about how your actions may be perceived by others.