SSOScan: Automated Testing of Web Applications for Single Sign

This paper is included in the Proceedings of the

23rd USENIX Security Symposium.

August 20–22, 2014 • San Diego, CA

ISBN 978-1-931971-15-7

Open access to the Proceedings of

the 23rd USENIX Security Symposium

is sponsored by USENIX

SSOScan: Automated Testing of Web Applications

for Single Sign-On Vulnerabilities

Yuchen Zhou and David Evans, University of Virginia

https://www.usenix.org/conference/usenixsecurity14/technic al-sessions/presentation/zhou

USENIX Association 23rd USENIX Security Symposium 495

SSOScan: Automated Testing of Web Applications for

Single Sign-On Vulnerabilities

Yuchen Zhou David Evans

University of Virginia

[yuchen, evans]@virginia.edu

http://SSOScan.org

Abstract

Correctly integrating third-party services into web ap-

plications is challenging, and mistakes can have grave

consequences when third-party services are used for

security-critical tasks such as authentication and autho-

rization. Developers often misunderstand integration re-

quirements and make critical mistakes when integrating

services such as single sign-on APIs. Since traditional

programming techniques are hard to apply to programs

running inside black-box web servers, we propose to de-

tect vulnerabilities by probing behaviors of the system.

This paper describes the design and implementation of

SSOScan, an automatic vulnerability checker for appli-

cations using Facebook Single Sign-On (SSO) APIs. We

used SSOScan to study the twenty thousand top-ranked

websites for ﬁve SSO vulnerabilities. Of the 1660 sites

in our study that employ Facebook SSO, over 20% were

found to suffer from at least one serious vulnerability.

1 Introduction

Single Sign-On (SSO) services are increasingly used to

implement authentication for modern applications. SSO-

enabled applications allow users to log into an applica-

tion using an established account (with a service such as

Facebook or Twitter) and connect their account on the

new site to an established Internet identity. Should the

application need more information from the user, it may

ask the user for extra permissions from the established

service. Once granted, the requested information is re-

turned to the application, which can then be used in the

transparent account registration process.

Although these services provide SDKs intended to en-

able developers without security expertise to integrate

their services, actually integrating security-critical third-

party services correctly can be difﬁcult. Wang et al. iden-

tiﬁed several ways applications integrating SSO SDKs

can be vulnerable to serious attacks even when develop-

ers closely follow the documentation [27].

To better understand and mitigate these risks, we de-

veloped SSOScan, an automated vulnerability checker

for applications using SSO. SSOScan takes a website

URL as input, determines if that site uses Facebook SSO,

and automatically signs into the site using Facebook test

accounts and completes the registration process when

necessary. Then, SSOScan simulates several attacks on

the site while observing the responses and monitoring

network trafﬁc to automatically determine if the appli-

cation is vulnerable to any of the tested vulnerabilities.

We focus only on Facebook SSO in this work, but our

approach could be used to check SSO integrations using

other identity providers or other protocols. Many of our

techniques could also be adapted to scan for vulnerabili-

ties in integrating other security-critical services such as

online payments and ﬁle sharing APIs.

1.1 Contributions

Our work makes two types of contributions: those related

to the construction of our scanning tool which are largely

independent of the particular vulnerabilities, and those

resulting from our large-scale study of Facebook SSO

implementations.

SSOScan. We explain the design and implementation of

SSOScan (Section 3), as well as how to handle some of

the challenges in the automation process. We describe

techniques that automatically perform user interactions

to walk through the SSO process (Section 3.1), includ-

ing clicking the correct buttons and ﬁlling in registration

forms. We collected information of almost 30,000 click

attempts for sites that implement Facebook SSO which

shows in detail how the individual heuristics are affecting

SSOScan’s behavior (Section 5.2). This provides exper-

imental evidence to support our design choices and shed

light on future research that shares a similar goal. SSO-

Scan can detect whether a target application contains any

of the ﬁve vulnerabilities listed in Section 2.2 with an

496 23rd USENIX Security Symposium USENIX Association

average testing time of 3.5 minutes, and is able to check

792 (81%) of the 973 websites that implement functional

Facebook SSO from the top 10,000 with no human inter-

vention at all.

Large-scale study. We ran SSOScan on the top 20,000

US websites (Section 4). Key results from the study in-

clude ﬁnding at least one vulnerability in 345 of the 1660

sites that use Facebook SSO (Section 4.1). We also learn

how vulnerability rates vary due to different ways of in-

tegrating Facebook SSO (Section 4.1.1). We manually

analyzed the 228 sites ranked in the top 10,000 that SSO-

Scan cannot test automatically and report on the reasons

for failures (Section 4.2). Our study reveals the complex-

ity of automatically interacting with web sites that follow

a myriad of designs, while suggesting techniques that

could improve future automated testing tools. In Sec-

tion 6, we discuss our experiences reporting the vulnera-

bilities to site owners and possible ways SSOScan could

be deployed.

2 Background

This section provides a brief introduction to single sign-

on systems, describes the vulnerabilities we checked, and

summarizes relevant previous work.

2.1 Single Sign-On

A typical single sign-on process involves three parties.

Alice ﬁrst visits a web application and elects to use SSO

to login. She is then redirected to the identity provider’s

SSO entry point (e.g., Facebook’s server). After she logs

into Facebook, her OAuth credentials are issued to the

application server. The application server conﬁrms the

identity and authenticates the client.

OAuth uses three different types of (rather confu-

singly-named) credentials:

Access

token. An access token represents permissions

granted by the user. For example, the application may

request that user grant permission to access the birth-

day and friend lists from her Facebook account. Upon

the user’s consent, a token will be issued and forwarded

to the application which may then use it to obtain the

granted information from Facebook. An access

token

eventually expires, but may be valid for a long time.

Code. A code is used to exchange for an access

token

through the identity provider. This exchange requires the

application’s unique app

secret to proceed. If the secret

does not match, Facebook will not issue the token. This

means a code is bound to a user as well as a target appli-

cation. With Facebook SSO, the code expires after being

used in the ﬁrst exchange.

Signed

request. A signed request is a base64 encoded

string that contains a user identity, a code, and a signa-

ture that can be veriﬁed using an application’s app

secret

and some other metainformation. Once issued, it is not

tied to Facebook (except for the enveloped code), and the

signature can be veriﬁed locally.

2.2 Vulnerabilities

Our interest in building an automatic scanning tool was

initially motivated by the access

token misuse vulnera-

bility reported by Wang et al. [27]. We further identiﬁed

four new vulnerabilities that are both serious and suitable

for automatic testing. The ﬁrst two vulnerabilities con-

cern confusions about how authentication and authoriza-

tion are done; the other three concern failures to protect

important secrets.

Access

token misuse. This vulnerability stems from

confusion about authentication and authorization. In

OAuth 2.0, an access

token is intended for authoriza-

tion purposes only because it is not tied to any speciﬁc

application. When a service uses an access

token to au-

thenticate users, it will also accept ones granted to any

other application. Figure 1 illustrates an impersonation

attack that exploits this vulnerability: Alice visits Mal-

lory’s website (step 1), logs in using Facebook SSO (2),

and receives an access

token from Facebook (3). Then,

Mallory’s client-side code running in Alice’s browser

forwards the access

token to Mallory (4), which presents

the token to a vulnerable application’s server (5). After

conﬁrming the token represents Alice, Foo’s application

server authenticates Mallory as Alice (6).

1. Visit

3. Issue credentials

4. Forward

credentials

5. Reuse credentials

6. Authenticated

Facebook

User

Mallory

Foo app server

2. Login

Figure 1: OAuth credential misuse

Signed

request misuse. Sometimes developers have

chosen the correct OAuth credentials to use, but still end

up with a vulnerable implementation. One way this hap-

pens is when information decoded from a signed

request

is used but the signature is never checked using the

app

secret. The attack to exploit this vulnerability is

USENIX Association 23rd USENIX Security Symposium 497

similar to the previous one, except that Mallory needs

to reuse the signed

request in addition to access token.

App

secret leak. When a developer registers an appli-

cation with Facebook, she receives an app

secret. It

is essential for the application owner to keep it a se-

cret because the app

secret is used as the key to cre-

ate signed

requests and to access many other privileged

functionalities. However, careless developers may reveal

this secret to clients, especially when using code ﬂow to

authenticate users. By design, the code and app

secret

must be sent from the application’s back end server to

Facebook in exchange for an access

token. When this

exchange is carried out through the client instead of the

server, app

secret is exposed to any malicious client.

User OAuth credentials leak. The last two vulnerabili-

ties both leak a user’s OAuth credentials. When the Face-

book OAuth landing page contains third-party content,

requests to retrieve those contents will automatically in-

clude OAuth credentials in the referer header, which

leaks them to the third-party. To thwart this leakage,

Facebook offers a layer of protection by only allowing

access

token and signed request to appear in the URL

fragments, which are not visible in the referer header.

Therefore only code can be leaked via referer unless the

application intentionally pulls the credentials and puts it

in the URL

. In addition, credentials can be exﬁltrated

by third-party scripts if they are present in the page con-

tent. If a malicious party is able to obtain these creden-

tials, it could carry out impersonation attacks or perform

malicious actions using permissions the user granted the

original application, such as posting on the user’s time-

line or accessing sensitive information. Note the differ-

ence between embedding OAuth credentials in the URL

and in the body content is that the former will directly

leak them to third parties, while the latter only leaks the

credential when the embedded third party code accesses

it explicitly.

2.3 Related Work

Our work builds on extensive previous work on automat-

ically testing applications for vulnerabilities. We brieﬂy

describe relevant approaches next, as well as previous

works that analyze vulnerabilities in SSO services.

Program analysis. Program analysis techniques such as

static analysis [3] and dynamic analysis including sym-

bolic execution [7, 17] automatically identify vulnera-

bilities with fast testing speed and good code coverage.

Runtime instrumentation techniques such as taint track-

ing [11] and inference [18] also help to safeguard sensi-

Surpsingly, we found several sites doing this (e.g., dealchicken.com

and bloglovin.com).

tive source-sink pairs. However, these techniques require

white-box access to the application (at least at the level of

its binary), which is not available for remote web appli-

cation testing. Automated web application testing tools

that work on the server implementation [1, 8, 16] do not

apply to large-scale vulnerability testing well. They ei-

ther require access to application source code or other

speciﬁc details such as UML or application states. For

our purposes, the test target (application server imple-

mentation) is only available as a black box.

Automated security testing. Penetration testing is

widely used to check applications for vulnerabilities [15,

28]. The tester analyzes the system and performs sim-

ulated attacks on it, often requiring substantial manual

effort. More automated testing requires an oracle to de-

termine whether or not a test failed. Sprenkle et al.

developed a difference metric by comparing two web-

pages based on DOM structure and n-grams [21] and im-

proved results using machine learning techniques [22].

SSOScan also requires an oracle (Section 3.2) to deter-

mine session identity. For our purposes, a targeted oracle

works better than their generic approach.

Automated GUI testing. SSOScan is also closely re-

lated to automated GUI testing. The GUI element trig-

gering approach we take shares some similarities with re-

cent works to simulate random user interactions on GUI

element to explore application execution space on An-

droid system [14], native Windows applications [29], and

web applications [5, 10]. Their common goal is to ex-

plore app execution space efﬁciently to discover buggy,

abnormal or malicious behavior. By contrast, our goal is

to drive the application through a particular SSO process

rather than explore its execution space. Further, we need

the tests to proceed fast enough for large-scale evalua-

tion. Since each simulated user interaction with the web

application involves round-trip trafﬁc and a non-trivial

delay to get the response, our primary focus is to develop

useful heuristics to quickly prune search space before

triggering any user interactions.

SmartDroid [32] and AppIntent [31] both aim to re-

cover sequences of UI events required to reach a par-

ticular program state or follow an execution path ob-

tained from static analysis. These approaches target An-

droid applications and rely on client-side information

that is not available for our web application scanning

tool, where the necessary state only exists on the (inac-

cessible) server side.

Human cooperative testing. Off-the-shelf testing tools

like Selenium [19] and TestingBot [24] can be used to

discover bugs in web applications under developers’ as-

sistance. These tools replay user interactions based on

testing scripts that are manually created by the applica-

498 23rd USENIX Security Symposium USENIX Association

tion developer. BugBuster [6] offers some automatic web

application exploration capabilities, but still does not un-

derstand the application context enough to perform any

non-trivial actions such as those involving authentication

and business logic.

To reduce developer effort, Pirolli et al. [13], Elbaum

et al. [9], and the Zaddach tool [12] show promising re-

sults by collecting interactions from normal users and re-

playing them to learn application states and invariants for

vulnerability scanning. These works do not require extra

manual effort from developers to write testing script or

specify user interactions. However, one potential prob-

lem these works fail to address is user’s privacy con-

cerns when submitting interactions. This could be es-

pecially sensitive when the actions involve passwords or

payments. SSOScan avoids this problem and is comple-

mentary to this line of work — SSOScan attempts to scan

applications in a fully automatic fashion and does not re-

quire traces from any party.

Single sign-on security. Single sign-on has emerged as

an important security service and has been well-studied

in recent years. Previous works have discovered prob-

lems in protocols, bugs in SDK code and missed assump-

tions in developers’ implementations [4, 20, 23, 25, 27].

Automated scanning is especially valuable for vulnera-

bilities that cannot be simply ﬁxed by upgrading SDKs

or improving the protocols, but stem from mistakes inte-

grating the SSO service.

Integuard [30] and AuthScan [2] have similar goals

with SSOScan. Integuard infers invariants across re-

quests and responses and uses them to perform intru-

sion detection on future activities. AuthScan [2] is an

automated tool to extract speciﬁcations from SSO imple-

mentations by using both static program analysis and dy-

namic behavior probing. Our goals differ in that we focus

on detecting speciﬁc vulnerabilities rather than generic

ones. This enables us to establish clear automation goals

and build well-deﬁned state machines for the scanner,

and removes the uncertainties the previous works incur

when inferring invariants or modeling unknown func-

tions. The drawback is our approach relies on knowledge

of particular vulnerabilities. For many integrated web

services, including SSO, many vulnerabilities are known

or can be obtained using systematic explication [27].

3 SSOScan

SSOScan consists of two main parts: the Enroller and

the Vulnerability Tester. The Enroller automatically reg-

isters two test accounts at a web application using Face-

book SSO. The Vulnerability Tester simulates attacks

and monitors trafﬁc to test for each vulnerability. In this

section, we describe the general workﬂow of these mod-

Vulnerability

Tester

Oracle

SSO

Button clicked

SSO

button finder

SSO process

automation

Registration

automation

SSO process

finished

No FB SSO

Detected

Homepage

Give up/

Manual analysis

Registration

successful

to try?

More strategies

See FB SSO

traffic?

Start

Yes

Figure 2: Enroller Overview.

Ovals represent testing states, curved rectangles represent different

modules in our tool, and diamonds represent control ﬂow decisions.

ules necessary to understand the results in Section 4, but

defer the details of our heuristics to Section 5.

3.1 Enroller

Figure 2 shows the workﬂow of the Enroller. Given a

target web application, our tool ﬁrst removes all cook-

ies from the browser and navigates to the target URL.

A short delay after the page has ﬁred its

onload event,

the SSO button ﬁnder (Section 3.1.1) analyzes the DOM

and outputs the most likely candidate elements for SSO

button. The Enroller then simulates clicks on those ele-

ments, monitoring trafﬁc to listen for the Facebook SSO

trafﬁc pattern. Once a click or sequence of clicks is found

that produces the recognizable SSO trafﬁc, SSOScan au-

tomatically logs into Facebook and grants the requested

permissions to the application.

About 44% of sites we tested still require a user to reg-

ister when using SSO, so it is important to automate this

process. SSOScan combines heuristics with random in-

puts to ﬁll in and submit the forms (Section 3.1.2), and

then uses an oracle (Section 3.2) to determine if the sub-

mission succeeds. If the oracle deems the registration to

be a failure, the Enroller tries using different strategies

(Section 5) until either the oracle passes or a threshold

level of effort is exceeded. The entire process succeeds

for 80% of the websites using Facebook SSO in the top

10,000 sites (Section 4 presents detailed results).

3.1.1 SSO Button Finder

A typical starting page, taken from

hufﬁngtonpost.com,

is shown in Figure 3. SSOScan needs to ﬁrst ﬁnd and

click the “Log in” button on the main page, and then the

“Log in with Facebook” button on the overlay that pops

up afterwards. As illustrated in Figure 4, SSOScan ﬁrst

extracts a list of qualifying elements from all nodes in

an HTML page, and then extracts content strings from

such elements. The Button Finder relies on the assump-

tion that developers put one of a small pre-deﬁned set

USENIX Association 23rd USENIX Security Symposium 499

of expected words in the text content or attributes of the

SSO button. It computes a score for each element by

matching its content with regular expressions such as

[ Ll ][ Oo][Gg][IiOo][Nn] which indicates its resemblance

to “login”. SSOScan forms a candidate pool consisting

of the top-scoring elements and triggers clicks on them.

(Section 5 describes the heuristic choices SSOScan uses

to ﬁlter elements and compute scores.)

3.1.2 Completing Registration

The required interactions to complete the registration

process after single sign-on vary signiﬁcantly across web

applications. They range from simply clicking a submit

button (e.g., Figure 5, in which all input ﬁelds are pre-

populated using information taken from the SSO pro-

cess), to very complicated registration processes that in-

volve interactively ﬁlling in multiple forms.

SSOScan attempts to complete all forms on the SSO

landing page by leaving pre-populated ﬁelds untouched

and processing the remaining inputs in the order of ra-

dios, selects, checkboxes and ﬁnally text inputs. We

found this ordering to be very important to achieve

higher automation success, as some forms may dynami-

cally change what needs to be ﬁlled upon selecting differ-

ent radio or select elements. Processing these elements

ﬁrst allows SSOScan to rescan for dynamically generated

ﬁelds and process them accordingly.

For radio and select elements, SSOScan randomly

chooses an option; for checkboxes, it simply checks

all of them. For text inputs, SSOScan tries to infer

their requirements using heuristics and provide satisfac-

tory mock values. Once all the inputs have been ﬁlled,

the next step is to reuse the SSO Button Finder (Sec-

tion 3.1.1) with different settings designed to ﬁnd submit

buttons. After SSOScan attempts to click on a submit

button candidate, it refers to the oracle to determine if

the entire registration process is successful.

Figure 3: SSO Buttons (hufﬁngtonpost.com)

String

Filtering

Extract

Content

Regex

matching

HTML

Element

Score

Figure 4: SSO button ﬁnder workﬂow

3.2 Oracle

The Oracle analyzes the application and determines

whether it is in an authenticated state, and if so, further

identiﬁes the session identity. This module is necessary

for SSOScan to decide if a registration attempt is suc-

cessful. It is also used by the Vulnerability Tester to de-

termine if a simulated impersonation attack succeeds.

The key observation behind the Oracle is that web ap-

plications normally remove the original login button and

display some identifying information about the user in

an authenticated session. For example, after a successful

registration many websites display a welcome message

that includes the user’s ﬁrst name.

After the page ﬁnishes loading, the Oracle searches

the entire DOM and

document.cookie for test account

user information (e.g., names, email, or proﬁle images).

We evaluate the correctness of our assumptions and ef-

fectiveness of our Oracle in Section 4.2.

3.3 Vulnerability Tester

After the Enroller successfully registers two test ac-

counts, control is passed to the Vulnerability Tester

which checks the target application for the vulnerabilities

described in Section 2.2. We use two different probing

approaches to cover the ﬁve tested vulnerabilities: simu-

lated attacks and passive monitoring.

Simulated Attacks.The two credential misuse vulnera-

bilities are tested using simulated impersonation attacks.

We describe how this is done for signed

request misuses;

the method for checking access

token misuses is similar.

Figure 5: Registration Form (espn.go.com).

500 23rd USENIX Security Symposium USENIX Association

To set up the tests, we created a test application Mal

which uses Facebook SSO, and obtained Alice’s sign-

request for Mal. This mimics the scenario where

Alice is tricked into visiting and signing into an arbi-

trary malicious website using Facebook. After the ac-

count registration ﬁnishes, we use Bob’s credentials to

sign into Facebook for target application, but replace the

signed

request in Facebook’s response with the prior re-

sponse received for Alice. For consistency, we also re-

place all access

tokens found in the trafﬁc.

The attack is successful if Bob is able to login as Al-

ice using the replayed signed

request. The Vulnerability

Tester deems the site vulnerable if the Oracle determines

that Alice is logged in after the simulated attack.

Passive Monitoring. The three credential leakage vul-

nerabilities are detected using passive approaches. For

brevity, we only explain how leaks through the referrer

header are detected; the other leaks are detected similarly

by observing network trafﬁc and web page contents.

To check if an application leaks the user’s OAuth cre-

dentials through the referrer header, SSOScan monitors

all request data during the account registration process

and compares each referrer header to OAuth credentials

recorded in earlier stages. If a match is found, SSOScan

then checks if the requesting page contains any third-

party content such as scripts, images, or other elements

that may generate an HTTP request. SSOScan reports a

potential leakage when credentials are found in the refer-

rer header for a page that contains third-party content.

4 Results

We evaluated SSOScan by running it on the list of the

most popular 20,000 websites based on US trafﬁc down-

loaded from

quantcast.com as of 7 July 2013. Of those

20,000 sites, 715 of the sites are shown as hidden proﬁle

(that is, no URL is given, thus excluded from our study).

We ran SSOScan on the remaining 19,285 sites in

September 2013, and found that homepages of 1372 sites

failed to load during two initial attempts (most likely due

to either expired or incorrect domain name, server error,

or downtime). We excluded these sites from our data set,

leaving a ﬁnal test dataset containing 17,913 sites.

Completing the tests took about three days, running 15

concurrent sessions spread across three machines. The

average time to test a site is 3.5 minutes. We limited the

maximum stalling time for each site on any one module

to four minutes, and the overall testing time to 25 min-

utes per site. If this timeout is reached, SSOScan restarts

and retries a second time before skipping it. We ran extra

rounds on tests that failed or stalled during initial round

until either the test is completed or the four rounds max-

imum limit has been reached. The extra rounds involved

Not Vulnerable

77.4%

Buggy

2.3%

Valid Top 20,000 sites

No Facebook

SSO, 83.1 %

Timeout/error 7.6%

Facebook

SSO, 9.3%

Vulnerable

20.3%

1660 Sites using Facebook SSO

Figure 6: Results overview

fewer sites (<10%) and took a week to complete running

on one machine with four concurrent sessions.

In July 2014, we re-ran the tests on the vulnerable sites

to see how many sites had corrected the vulnerabilities.

The results from that scan are reported in Section 6.2.

4.1 Automated Test Results

Figure 6 presents results purely based on automatic tests

run by SSOScan. SSOScan found a total of 1660 sites

using Facebook SSO among the 17,913 sites (9.3% of

the total). Figure 7 shows the number of Facebook SSO

supported sites, sites that misuse credentials, and sites

that leak credentials distributed by site ranking. The dot-

ted lines on top of the bars show the average stats of all

sites that are more popular than that rank. In Section 4.3,

we report on our manual analysis on failed tests for sites

ranked in the top 10,000.

Facebook SSO integration. Figure 7 (a) shows that

more popular sites are more likely to integrate Facebook

SSO. Of the top 1000 sites, 270 (27%) of them include

Facebook SSO, compared to only 52 out of the 1000

lowest-ranked sites in our dataset. This supports our be-

lieve that covering the top-ranked 20,000 websites is suf-

ﬁcient to get a clear picture of prevailing Facebook SSO

usage since less popular sites are both less visited and

less likely to use Facebook SSO.

Faulty implementations. To implement Facebook SSO,

an application must be conﬁgured correctly in the Face-

book developer center. Using incorrect parameters to call

the SSO entry point also result in errors that will prevent

any user from authenticating to that application through

SSO. Such cases, automatically identiﬁed by SSOScan,

were more common than we expected. The most popular

errors include setting the application to ‘sandbox’ mode

(for development stage only) in the developer center, or

providing a wrong application ID. SSOScan found 39

(2.3% out of 1660 sites that incorporate Facebook SSO

buttons) sites that display visible Facebook SSO buttons

but have implementations so buggy that no user could

ever login using them. A possible explanation is that the

buttons are there for SEO purposes and the developers

USENIX Association 23rd USENIX Security Symposium 501

10%

15%

20%

25%

30%

35%

40%

45%

% supporting FB SSO

10%

15%

20%

25%

30%

35%

40%

45%

% vulnerable

10%

15%

20%

25%

30%

35%

40%

45%

% vulnerable

(a) Facebook SSO support

Site rank (100 equally sized buckets, each containing 1% (179) of all valid test sites)

1 10 20 30 40 50 60 70 80 90 100

Each bucket represents all sites using Facebook SSO that are ranked in the corresponding range in (a)

Figure 7: Facebook integration results by site rank

never actually bothered to implement it, or the develop-

ers simply copied and pasted an SSO snippet customized

for another application without ever testing it.

Vulnerability trends. We found 202 sites (12.1%) that

misuse credentials (126 of which are misusing both ac-

cess

token and signed request) and 146 sites (8.6%) that

leak Facebook SSO credentials (of which 72 sites are

leaking through both referrer headers and DOM). A to-

tal of 345 sites (20.3%) suffered from at least one of the

ﬁve tested vulnerabilities, and 3 sites suffered from both

credential misuse and leakage problems.

It is also worth noting that SSOScan did not ﬁnd any

sites leaking their app

secret to the public by calling the

token exchange API on the client side. To verify that

we implemented the check correctly, we have conﬁrmed

that SSOScan does correctly identify this vulnerability

on our manually-crafted faulty application. This is an in-

teresting result, especially compared to the high number

of sites that have at least one of the other vulnerabilities.

We suspect this is partly due to explicit warnings in the

documentation and the increased effort required to actu-

ally implement the token exchange on the client side.

As shown in Figure 7 (b) and (c), more popular sites

appear to be more likely to have credential misuse vul-

nerabilities, while less popular sites tend to have more

credential leakage problems. This fact certainly raises

concern — credential leakage could potentially do dam-

age to users’ Facebook accounts, and it would be hard to

contact numerous low-proﬁle problematic sites to have

them all ﬁxed. The victim’s Facebook account is in jeop-

ardy if any of the applications he or she uses have such

problem. Even though credential misuse cannot harm

Facebook accounts directly, the fact that such vulnera-

bilities exist in high-proﬁle websites is worrisome, as im-

personation attacks carried against sites with millions of

users have more severe consequences thank similar at-

tacks on lower-proﬁle sites.

Of the top-1000-ranked sites, 60 of the 270 (22.2%)

that support Facebook SSO are found to have at least one

vulnerability. The vulnerability rate is 21.3% across all

sites in the top 10,000 and 18.5% for sites ranked from

10,001 to 20,000. This overal vulnerability rate suggests

that development practices at larger companies do not ap-

pear to be more stringent (at least with respect to SSO)

than they are at less popular sites.

As we do not have access to server side source code,

we cannot measure how reusing code may positively or

negatively affect the vulnerability trend. However, we

did notice that some sites use fourth party services (e.g.

Janrain, Gigya) to implement the Facebook SSO. In such

scenarios, the user effectively does two SSO processes

during authentication — the user, Facebook (IdP) and

Janrain (RP) initially; the user, Janrain (IdP) and the true

relying party afterwards. As the Facebook SSO process

is entirely handled by the fourth party and is hidden to the

relying party, the RP’s behavior is not relevant to this vul-

nerability. We have manually tested both Janrain and Gi-

gya’s Facebook SSO implementation for credential mis-

use vulnerabilities and conﬁrmed that both of them cor-

rectly implement the process by only using code ﬂow to

authenticate users. As a result, sites using these services

contribute to a lower vulnerability rate. Note that the RP

would still need to implement the second SSO process

correctly to avoid vulnerabilties, but SSOScan currently

does not check IdPs other than Facebook.

502 23rd USENIX Security Symposium USENIX Association

4.1.1 Front-end Integration

There are three basic client-side methods to integrate

Facebook SSO: a JavaScript SDK, a pre-conﬁgured wid-

get, or a custom implementation. (We have no way to

determine how the developers are integrating Facebook

SSO at the back end.) We used SSOScan to aggregate

front-end integration choices and compare them with

vulnerability reports. Table 1 summarizes the results.

Websites using client side SDKs and pre-conﬁgured wid-

gets are more likely to misuse credentials (29.1% and

15.5% vs. 1.3% in non SDK/widget implementations).

Our guess is that this is due to the way SDKs and widgets

conveniently expose raw access

token, signed request,

or even user name Facebook ID values. This convenience

may lead to the developers to neglect to check the signa-

ture and the intended audience of the credential. How-

ever, our results also show that websites using SDKs and

widgets are better in hiding credentials (3.6% and 2.2%

compared to 12.4% vulnerable rate in SDK/widget im-

plementations). This is likely because such applications

use the Facebook-provided landing page which has safe

redirect URLs and no third-party content. Applications

built this way are secure unless the developers explicitly

add the credentials in the page content or URL.

4.1.2 Examples

We describe two examples of vulnerabilities found by

SSOScan here to illustrate the potential risks. Section 6

discusses our experiences reporting vulnerabilities to site

owners and Facebook.

Match.com. Ranked 118

on the list, Match.com is a

popular online dating website. SSOScan revealed that

match.com is also vulnerable to signed

request replace-

ment attacks. To use

match.com services, users need

to provide sensitive information including their birthday,

location, photos, personal interests, and sexual orienta-

tion. Impersonators will not only have access to this in-

formation, but also learn whom the victim is dating and

possibly the time and location of the dates.

Fodors.com. Fodor’s is a travel advice website that is

the 217

-ranked US site. Its redirection landing page

contains access

token information along with some other

Method Number Misuse Leakage

SDK 578 29.1% 3.6%

Widget 132 15.5% 2.2%

Custom Code 950 1.3% 12.4%

All 1660 12.1% 8.6%

Table 1: Rate of credential misuse and credential leakage

for different Facebook SSO front-end implementations

third-party scripts in its content. The scripts come from

various sources including

quantserve.com, fonts.com,

yahooapis.com, and multiple domains owned by Google.

The permission Fodor’s requests includes user’s basic in-

formation, email address, and more importantly, permis-

sion to post to user’s wall on the his or her behalf. This

means if the access

token is leaked to a malicious party,

it can post to a user’s Facebook wall without consent in

addition to accessing the user’s basic information.

4.2 Detection Accuracy

To evaluate the detection accuracy of SSOScan, we sam-

pled test cases from all results (including sites reported

to have no Facebook SSO support, secure and vulnera-

ble cases) and manually examined them. We consider

two types of mistakes: misreporting whether the site

integrates Facebook SSO, and incorrectly determining

whether or not a Facebook SSO-enabled website exhibits

a vulnerability.

Facebook Login Detection Correctness. SSOScan

searches SSO button based on heuristics and cannot

guarantee success for all websites. Indeed, it is not possi-

ble for anyone to determine with complete conﬁdence if

a website uses Facebook SSO by just browsing the site.

To roughly measure how many Facebook SSO-enabled

websites were missed by SSOScan, we randomly sam-

pled the 100 sites that were reported by SSOScan to have

no Facebook SSO support and manually examined them.

To make the samples representative of the whole set, we

picked one site out of every 200 sites ordered by their

rank. From manually investigating these 100 sites, we

could only ﬁnd one site that included Facebook SSO but

was missed by SSOScan. As we introduce later in Sec-

tion 6, we also deployed SSOScan as a web service that

is made available to use in our research group. The web

service has received a total of 69 valid submissions so far

and we have also manually examined the vulnerability

reports.

We found four cases (5.8%) where a submitted

site included Facebook SSO but SSOScan was not able

to trigger it.

The sites that SSOScan fails to ﬁnd Facebook lo-

gin present unusual interfaces which our heuristics are

not able to navigate to. Speciﬁcally,

oovoo.com and

bitdefender.com do not show any login button on its

homepage, but instead the user needs to click a ‘my ac-

count’ button to initiate the login process. The

sears.com

site displays a login button on its homepage, but the SSO

process is not initiated until the user interacts with the

popup window three times, which exceeds the maximum

These have mostly been sites suggested by people we have demoed

SSOScan scan to, since the service has not yet been publicized. Hence,

it is a small and non-representative sample, so not clear what we can

conclude from this at this point.

USENIX Association 23rd USENIX Security Symposium 503

click depths (two) in this evaluation. We have also seen

one case (

coursesmart.com) in which the login process is

rather typical, but SSOScan still missed the correct login

button (that button is scored the 4th highest while SSO-

Scan only attempts to click the top 3 candidates.). Most

of these issues may be addressed with more relaxed re-

strictions and more regular expression matching as de-

scribed in Section 5.2. Finally, our prototype implemen-

tation is limited to English-language websites due to its

string matching algorithm, but could be extended to in-

clude keywords in other languages.

SSOScan may also incorrectly conclude that a web-

site supports Facebook SSO when it does not. We

have seen sites (e.g.,

msn.com) that only use Facebook

SSO to download user activities and display them on

the page, but do not integrate their identity system with

Facebook SSO. Although SSOScan is designed to skip

searching on typical Facebook-provided social plugins

and widgets, non-standard integration of such function-

alities may rarely lead to false positives.

Vulnerability Status Correctness. Since SSOScan sim-

ulates potential attacks and veriﬁes their success or fail-

ure, detection is likely to be highly accurate. Never-

theless, we consider several possible reasons that might

cause false positives/negatives to be reported.

SSOScan should be able to capture all credential leak-

age vulnerabilities with no false positives. A false neg-

ative may occur since SSOScan only looks for exact

matches to the original OAuth credential string, so it will

not report a leakage if the credential is slightly trans-

formed or encoded. Further, SSOScan only observes

trafﬁc involving the web client, so does not detect appli-

cation that leak OAuth credentials outside the SSO pro-

cess.

SSOScan only reports a credential misuse vulnerabil-

ity when it can successfully execute an impersonation

attack. So, the only risk for incorrect reports is if the

Oracle incorrectly determines the session identity. We

designed the Oracle to minimize this risk. For example,

information for the test account is chosen carefully to

be unlikely to appear otherwise but to be close enough

to real names to pass sanity checks. For example, the

randomly generated name “Syxvq Ldswpk” was rejected

by a small number of websites, but “Jiskamesda Qua-

narista” always passed sanity checks and only appeared

in an authenticated session in all of our tests. Barring an

unlikely name collision, there does not appear to be any

way SSOScan would produce a false positive credential

misuse report.

The Oracle checks the whole response for identifying

information instead of only the DOM content to han-

dle sites which only embed such information in ﬁrst-

party cookies after logging in. In some rare cases, these

cookies could be issued even before SSOScan ﬁnishes

registration forms. This means that before the Enroller

searches for registration forms to ﬁll in, the Oracle deems

registration as unnecessary because it concludes that the

application is already in an authenticated state. Although

SSOScan is able to proceed and determine vulnerability

status, the application never enters an authenticated state

and false negatives might occur.

Trusted Third-Party Domains. For credential leakage

vulnerabilities, SSOScan reports an application as vul-

nerable if it identiﬁes visible credentials co-existing with

any content or script that comes from any origin other

than the host or Facebook. This could overestimate the

vulnerable sites because the host may own other domains

and serve content over them, which should not be consid-

ered untrusted. For example, content delivery networks

and sub-company scenarios (e.g.,

cnn.com embedding

content from

turner.com which owns CNN) are common

among popular websites.

4.3 Automation Failures

For about 19% of the top 10,000 tested site that include

functional Facebook SSO, SSOScan is not able to fully

automate the checking process. Figure 8 shows the dis-

tribution of rank of failed test websites.

To better understand the reasons why SSOScan fails,

we manually studied all 228 failed cases reported by

SSOScan for sites ranked in the top 10,000. We found

that although 47 out of these 228 cases set their Face-

book application conﬁgurations and SSO entry points

properly, they never respond to credentials returned by

Facebook SSO, which means no users would be able to

successfully log into these sites through Facebook SSO.

Excluding these 47 left us a total of 181 failure cases.

Registration automation failure. By far the most com-

mon reason for SSOScan to fail is due to complicated or

highly-customized registration process. We found 43.7%

10%

15%

20%

25%

30%

35%

40%

45%

% failed

Percentage of failed tests vs. Site rank

Overall:

19%

1 10 20 30 40 50 60 70 80 90 100

Site rank (100 equally sized bucket)

Figure 8: Failed tests rank distribution

504 23rd USENIX Security Symposium USENIX Association

Failure reason Number Percent

linking/subscription 51 28.1%

CAPTCHAs 34 18.8%

identity invisible to oracle 28 15.5%

atypical input elements 20 11.0%

atypical submit buttons 19 10.5%

email veriﬁcation 10 5.5%

non-HTTPS submission forms 9 5.0%

other (e.g., timeouts) 10 5.5%

Total failures 181 100.0%

Table 2: Automation Failure Causes (top 10,000 sites)

of the sites that implement Facebook SSO still require

users to perform additional actions to complete the reg-

istration (roughly evenly distributed by site popularity).

SSOScan failed to complete registration on 143 (33.6%)

of them. Table 2 shows the major reasons contributing

to this failure ordered by their occurrences: 1) sites that

require SSO users to link to an existing account or pro-

vide payment information to subscribe to the service;

Currently SSOScan cannot handle the “linking” action:

automatically registering a “traditional” account and per-

form the linking poses an out-of-scope challenge — do-

ing so often requires solving CAPTCHAs

. 2) registra-

tion forms after the SSO process include CAPTCHAs;

3) special input elements (e.g.

div, span or image as op-

posed to

input) cannot be found automatically, or spe-

cial requirements for the input that cannot be fulﬁlled;

4) sites where the registration submission button cannot

be located; 5) sites that requires users to conﬁrm email

addresses before continuing (usually this involves click-

ing a link in an email sent by the server to the user’s

email address); and 6) sites that insecurely send regis-

tration data using a non-HTTPS form which causes the

testing browser to pop up a warning and stall.

Oracle confusion. SSOScan may also fail because the

oracle reports failure (15.5%), which occurs when it de-

tects the login button no longer exists after Facebook

SSO but cannot identify the session identity. We man-

ually analyzed such cases and found the biggest obstacle

is that the application homepage does not include any

identifying information at all. For example, instead of

showing ‘Welcome, {username}’, it shows ‘Welcome,

customer’, or simply ‘Welcome’, and the user name is

only displayed when accessing the account information

On the contrary, most tested applications (942 out of 973, see Sec-

tion 4.3) do not ask users to solve CAPTCHAs when an account is

created through SSO. This is a reasonable practice, since the user who

is able to provide a valid Facebook account should have already passed

Facebook’s requirements, and adding additional CAPTCHAs would be

unnecessarily annoying to the users.

page. In other cases, SSO authentication serves only a

sub-service of the website such as its afﬁliated forum,

but not the homepage which does not display any identi-

fying information.

Others. During the testing, we have also seen a number

of sites with extremely long loading time or inconsistent

network latencies after Facebook SSO or upon navigat-

ing to certain pages. While the latency spikes can likely

be resolved by re-running the tests, frequent long delays

which accumulate to SSOScan’s maximum timeout will

always halt the automation process. For example, this

happens when SSOScan accidentally triggers a browser

conﬁrmation dialog that requires user interaction, or ask-

ing users to stop a busy script execution.

5 Heuristics Evaluation

The ability of SSOScan to successfully complete the

Facebook single sign-on and registration process de-

pends on heuristics it uses to ﬁnd buttons and ﬁll in regis-

tration forms. Since each attempted button click involves

a high-latency round-trip with the server, early pruning

of search space and prioritization of elements is impor-

tant for achieving successful completion within a reason-

able amount of time. This section describes and analyzes

the heuristics SSOScan uses. We analyze the click data

collected from the top 10,000 sites that use Facebook

SSO and show how tweaking the heuristics signiﬁcantly

improves performance.

5.1 Options

Each step in the automation process can be controlled

by many options, including ﬁlters that can be enabled to

eliminate candidate elements that are unlikely to be the

correct target, weightings that adjust the contribution of

different element properties to its score, and other behav-

ior modiﬁers. The ones SSOScan used when running the

Section 4 study are described below; additional options

are described in our tech report [33].

Candidate rank. The button ﬁnder produces a candidate

element list ranked by score. SSOScan will ﬁrst attempt

clicking on the highest-ranked element, but sometimes

Figure 9: Example corner cases

USENIX Association 23rd USENIX Security Symposium 505

the correct element is ranked lower. This option controls

the maximum number of click attempts SSOScan makes

before succeeding or giving up. For Section 4’s experi-

ment, the lowest ranked element SSOScan clicks is the

third.

Visibility ﬁlter. Most websites only expect users to click

on UI elements that are visible, so the button ﬁnder in-

cludes a ﬁlter that ignores all invisible elements (e.g., el-

ements with zero height or width, shadowed in the back-

ground layer, or those which appear only when the user

scrolls the initial screen position).

Position ﬁlter. We noticed that SSOScan sometimes gets

distracted by a search box submit button when complet-

ing the registration form, even if it is able to correctly

ﬁll in the required information in all input elements. To

eliminate these misclicks, the position ﬁlter eliminates

the submit buttons which are displayed above any inputs

based on our observation that submit buttons nearly al-

ways come last in a registration form.

Registration form ﬁlter. As mentioned earlier in Sec-

tion 4.3, many websites provide two actions for the user

after SSO is completed: ‘create new account’ or ‘link an

existing account’. The latter option requires the user to

enter the user name and password of an existing account

to ﬁnish the enrollment process. To avoid these, the reg-

istration form ﬁlter rejects a candidate submit button if

its parent form contains only two visible text inputs, one

has the meaning of ‘name’ or ‘email’ and the other is of

type password, since such an element is most likely to be

a submit button of a linking form.

Element content matching. SSOScan searches for ele-

ments whose labels are close to “login with Facebook”

for SSO buttons by default. However, quite a few pop-

ular websites (e.g.

coupons.com, right side of Figure 9)

only allow users to “sign up with Facebook” ﬁrst before

logging in with Facebook. If the user has yet to do this,

attempting to login with Facebook will produce an er-

ror. To handle this situation, SSOScan will search for

elements with semantics similar to “sign up with Face-

book” when it fails to register using the “login” buttons.

A ﬁlter may signiﬁcantly reduce the number of mis-

clicks. However, it may also occasionally exclude cor-

rect elements. For example, not every correct submit

button is below all inputs (e.g., left of Figure 9, and

expedia.com’s submit button would have been missed

with the element position ﬁlter enabled).

Hence, SSOScan is designed to explore target sites us-

ing different option settings if enrollment does not suc-

ceed with the initial settings. It will continue to attempt

to complete the enrollment process using different set-

tings until either all conﬁgurations have been exhausted

or the timeout threshold is reached. SSOScan avoids do-

ing duplicate work by detecting if a click attempt has

resulted in a previously visited or completely explored

state (see our tech report for details [33]).

5.2 Experiment Setup

In theory, SSOScan could exhaustively trigger clicks on

every element on the page (and on all response pages up

to some maximum depth), which would result in nearly

100% success rate. This would be prohibitively slow in

practice, though, so the number of attempted clicks must

be limited for any realistic test. Given the time needed

for each click attempt, it is important to conﬁgure our

scoring heuristics well to maximize the probability of a

successful enrollment in the minimum amount of time.

To gather statistics about the candidate elements, we

modiﬁed SSOScan to try all possible strategies even if it

has already found the correct login button and to record

information about all attempted clicks, including for ex-

ample their size, position, visibility to the user, content

string feature and whether it is successful. We deﬁne a

click as successful if it is included in any sequence of

clicks from the start page to triggering the SSO process,

regardless of whether it appeared in an attempt that failed

to trigger the process. Because SSOScan skips previ-

ously explored states to avoid redundant effort, it auto-

matically rejects click sequences which involves cyclic

state transitions such as clicking on an irrelevant link and

then clicking on a logo which returns to the initial state.

We set up SSOScan to expand the candidate pool size

for each conﬁguration from 3 to 8, add more matching

regular expressions (e.g., to match the string “forum”

which occasionally leads to a login page on sites where

no login is visible on the start page), and use equal weight

for each of them. We also removed all candidate ﬁlters

described in Section 5.1. Our goal is to capture as many

ways to trigger the SSO process as possible by doing

as close to an exhaustive search as is feasible. This in-

creases the time required to scan a typical site to almost

an hour (compared to a few minutes with the setup used

in the full study).

We ran the test on the 973 sites from the top-10,000

ranked sites that were detected by SSOScan to support

Facebook SSO in our main study (Section 4). This bi-

ases the study slightly, since it only includes sites where

the conﬁguration used in the initial study was able to

ﬁnd Facebook SSO. Ideally we would like to run all top-

10,000 sites to avoid any bias introduced by the data set,

but the signiﬁcantly increased testing time prohibits us to

do so, and the result of our subsequent study on a random

sample of sites (Section 5.4) supports the claim that only

few sites containing Facebook SSO were missed by the

main study.

506 23rd USENIX Security Symposium USENIX Association

5.3 Results

The experiment recorded 29,539 unique

click attempts,

of which 5086 (17.2%) are successful (that is, they either

directly trigger SSO, or lead to subsequent clicks that

trigger SSO). This amounts to approximately 30 unique

clicks attempted per site, but the number varies signiﬁ-

cantly based on the site design, from a few up to 109.

Element type and content. Figure 10 shows how dif-

ferent button types and properties impact success rates.

We report the success rate as the number of times that

element appeared as a successful click divided by the to-

tal number of clicks attempted on elements of that type.

The number beneath the element feature gives the total

number of times that type of element occurred as a suc-

cessful click target across all the test sites. For exam-

ple, the BUTTON element type has an excellent success

rate — 60% of all BUTTON candidates are true positives

for the Facebook SSO button. But since it only appears

as a successful click on 78 out of 973 sites in our sam-

ple, it is rarely useful. By contrast, clicking on DIV ele-

ments are much less likely to trigger the Facebook login,

but such elements are more common. The right side of

Figure 10 shows that elements that are directly visible

to the user has a higher success rate than invisible ones,

and elements residing in iframes are twice as likely to be

the correct target as their counterparts in the main page.

These results suggest ways of weighting element types to

improve the scoring function and increase the likelihood

of ﬁnding successful clicks early.

If two clicks happens on pages with the same URL, same element

XPath and same element outerHTML, we consider them the same click.

True positives / all clicks

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Type: A DIV SPAN IMG BTN INPUT IFRM LI

visible invisible main iframe

Occurrences as

successful clicks

1556 266 204 188 78 64 9 1 1223 1144 2330 37

properties:

Figure 10: Login button type statistics

Content:

True positives / all clicks

0.1

0.2

0.3

0.4

0.5

0.6

logion Facebook FB signion account connect forum oauth

2816 1398 1196 896 422 297 218 74

Occurrences as

successful clicks

Figure 11: Login button content statistics

Pixels

100

200

300

400

Width, FP

Width, TP

Height, FP

Height, TP

0 10 20 30 40 50 60 70 80 90 99

Percentile

Figure 12: Impact of Login Button Size

Figure 11 shows how the success rate varies with at-

tribute content (matched by the given regular expres-

sion). The keyword “oauth” rarely exists in any content,

but when it appears it is very likely to identify the target

element. The result also shows that “FB” is not a good

indicator to predict the target, and we think this is proba-

bly because it is very short and may be used for similarly

named JavaScript variables or random abbreviations.

Both ﬁgures include data for the ﬁrst click only (but do

measure ﬁrst click success based on subsequent clicks).

Data for the second clicks are noticeably different from

the ﬁrst, and overall success rates are lower on second

clicks. The most interesting fact we found is that “con-

nect” (39%) and “Facebook” (36%) become the most

successful matches of all regular expressions, followed

by “oauth” at (26%). No other regular expressions ex-

ceed 20% success for the second click.

Element size. Figure 12 gives the cumulative distribu-

tion function of the width and height of target elements.

For example, the 80

percentile width of the true positive

elements is approximately 150px, compared to 300px for

false positive elements. We did not ﬁnd any signiﬁcant

difference between ﬁrst and second clicks, so the ﬁgure

combines data from all clicks. The key result is that wide

elements are less likely to be true positives, possibly due

to SSOScan incorrectly including many large underlay

elements as candidates. The result is similar for element

height (the lower two lines in the ﬁgure). This suggests

that it would be useful to add a ﬁlter function that ex-

cludes candidates whose width is greater than 300px. We

would expect it to eliminate 20% of the false positives

while hardly missing any of the true positives. Alterna-

tively, SSOScan could adjust the ﬁnal score of a node

according to its size based on these results.

Element position. Figure 13 shows the heatmap of the

cation indicates the number of elements found there sat-

isfying the property. Only visible elements are shown,

and each successful click only attributes to the intensity

once. All four ﬁgures are normalized with respect to their

maximum intensity (i.e., element density).

The ﬁgures show an interesting distinction from ﬁrst

click to second click: successful ﬁrst clicks almost ex-

USENIX Association 23rd USENIX Security Symposium 507

First Click, True Positive First Click, False Positive

Second Click, False Positive Second Click, True Positive

Figure 13: Login button location heatmap

clusively appear in the upper right corner of the page,

while the second click appears generally in the upper-

middle part of a page. The false positives are relatively

more scattered everywhere on the page

. This result sug-

gest we should assign a higher weight for elements for

these locations, and focus on elements in the vicinity of

the upper right corner for the ﬁrst click. We could po-

tentially even ignore the other criteria and only consider

position to ﬁnd login buttons on foreign-language sites.

5.4 Validation

After incorporating what we learned from these results

(e.g., weight adjustment for different button sizes and

types), we reran the SSOScan with the new heuristics

on the sites ranked from 10,000 to 20,000 that SSOScan

determined to support Facebook in the original study,

which were not included in the heuristics evaluation. We

compare the results with those obtained by using a “con-

trol” version of SSOScan, with equal weights on all fea-

tures and no candidate ﬁltering. All other settings such as

candidate pool size are the same between two versions.

The results support the hypothesis that adjusting heu-

ristics according to the results of the evaluation can im-

prove the speed and robustness of detection of Facebook

SSO integrations. The na

ıve control version missed 72

out of the 601 sites while the new heuristics missed only

two. The average rank of correct candidate elements for

the ﬁrst and second click is 1.32 and 1.23 for the con-

trol experiment, which improves to 1.23 and 1.17 respec-

tively with the new heuristics.

We also randomly picked 500 random sites from the

sites that SSOScan have yet to ﬁnd Facebook support

in the experiment in Section 4. We tested the expanded

The ﬁgures also show a clear width boundary. In the experiments

the browser resolution is 1920x1200, and it seems that most develop-

ers’ designs follow a standard width of approximately 960px, which is

why the density appears to be cut off.

heuristics on these sites, and further increased the max-

imum click depth to three to see if more SSO integra-

tions could be found. Individual tests took an average of

31 minutes to ﬁnish, but varies signiﬁcantly from a few

minutes up to an hour (threshold) based on site content.

Four additional sites were found that support Face-

book SSO from this sample in total. Two are found due to

the added regular expression

[Ff ][ Oo][Rr][Uu][Mm], one

of which required three clicks to trigger the SSO process.

Another site is found due to the improved candidate rank-

ing algorithm, and the fourth was found using the new

candidate selection method that includes all elements in

the right corner of the page, even if they do not match

any regular expressions. This provides a reasonable de-

gree of conﬁdence that our original study found a large

enough fraction of all the popular sites using Facebook

SSO to be representative, although likely missed around

1% of Facebook SSO sites. We did not try click depths

greater than 3 because of the exponential time growth re-

quired to complete each test, but we feel conﬁdent that

the number of Facebook SSO interfaces that can only be

discovered by attempting more than 3 clicks is very low.

6 Discussion

This section concludes by discussing limitations of SSO-

Scan, sharing our experiences reporting vulnerabilities,

and suggesting ways SSOScan can be deployed to help

secure applications integrating SSO services.

6.1 Limitations

While SSOScan is able to automatically synthesize ba-

sic user interactions and analyze trafﬁc patterns, this ap-

proach is not suitable for detecting all types of vulner-

abilities. It only works for vulnerabilities that can be

checked by observing trafﬁc or simulating predictable

user events, and falls short if the vulnerability testing in-

volves deep server-side application scanning or compli-

cated interactions. For example, Wang et al. [27] point

out that the application’s app

secret might be leaked to

arbitrary party if any page including Facebook’s PHP

SDK invokes two functions in a speciﬁc way. This

type of vulnerability could be checked at the developer

side using program analysis techniques, but cannot be

checked by an external tool with no awareness of the

sites’ implementation details or internal state.

6.2 Communication and Responses

We started contacting the site owners shortly after obtain-

ing our ﬁrst list of vulnerable sites, manually sending out

notiﬁcations to 20 vulnerable websites that we thought

were interesting. We contacted them either by submitting

508 23rd USENIX Security Symposium USENIX Association

a form on their website or through email. The responses

were very disappointing, especially compared with our

previous experiences reporting SDK-level vulnerabilities

to identity providers who tend to respond quickly and ef-

fectively to vulnerability reports [27]. The vulnerabili-

ties found by SSOScan, on the other hand, are primar-

ily in consumer-oriented sites without dedicated security

teams or clear ways to effectively report security issues.

Of the 20 notiﬁcations, we only received eight re-

sponses, most of which appear to be automated. After the

initial response, three websites sent us follow-up status

updates.

ESPN.com thanked us and told us the message

has been passed onto appropriate personnel, but no fol-

low up actions ensued. One of

answers.com’s engineers

asked us for more details, but failed to respond again af-

ter we replied with proposed ﬁx. As of July 2014, both

sites are still vulnerable. Four months after getting the

automated reply from

ehow.com, we received a response

stating that they have removed Facebook SSO from their

website due to “content deemed inappropriate”, and we

have conﬁrmed that the Facebook SSO button has in-

deed been removed. Sadly, we think their staff likely

did not (bother to) understand our explanation for the ﬁx

and simply removed the feature.

The other instance where a reported vulnerability was

ﬁxed was for

hipmunk.com. Hipmunk was found to be

vulnerable to both the access

token and signed request

replacement attacks. We did not get any response from

Hipmunk when the vulnerability was reported through

the normal channels, but through a personal connection

we were able to contact them directly. This led to a quick

response and series of emails with one of Hipmunk’s en-

gineers. We explained how to check the signature of

a signed

request, which should ﬁx both vulnerabilities.

However, when they got back to us believing that the ﬁx

was complete, we re-ran SSOScan and found that Hip-

munk was still vulnerable to the access

token replace-

ment attack. This meant Hipmunk checked the signa-

ture of signed

request after the ﬁx, but never decoded the

signed message body and compared its Facebook ID with

the one returned by exchanging access

token. This sur-

prised us, as we implicitly assumed the developers will

consume the signed message body after verifying its sig-

nature, and thus only included ‘verifying signature’ in

the proposed ﬁx. After further explanation, the site was

ﬁxed and now passes all our tests.

Retesting vulnerable sites. We retested all 345 vulnera-

ble sites in May 2014, nine months after our initial exper-

iment, including the 20 websites we had notiﬁed directly.

SSOScan found that 48 of the sites had eliminated the

vulnerabilities (including one out of the 20 sites we con-

tacted,

mapquest.com). Of the 48 ﬁxed sites, 22 had pre-

viously been diagnosed as credential leaking sites, and

27 were misusing credentials (one site,

trove.com, ﬁxed

both problems). We further examined these sites man-

ually to investigate the possible reasons and measures

to ﬁx the problems. As for sites that ﬁxed credential

misuses, we found that many had abandoned the token

or signed

request ﬂow in favor of the more secure code

ﬂow, which automatically protects them from credential

reuse attacks. For credential leakages, we found that

a number of sites redesigned their SSO process to fea-

ture a smoother user experience, e.g., replaced traditional

redirection ﬂows with AJAX operations, which naturally

eliminated credential leakage via referer header.

Communication with Facebook. Due to the ineffec-

tiveness of our direct communication with site owners,

we contacted Facebook’s platform integrity team in May

2014. Facebook’s engineers indicated that they are par-

ticularly worried about access

token leakage through ref-

erer headers (because a malicious party in possess of the

token may perform privileged Facebook actions on be-

half of the user, which potentially directly harms Face-

book), but are also concerned with the credential misuse

scenario. Facebook asked for a list of the vulnerable ap-

plications and contacted all the sites with access

token

leakage and credential misuse vulnerabilities (a total of

95 sites that we were able to re-conﬁrm at the time of re-

port), and informed us that they would “take enforcement

action as necessary” upon the ten sites that are leaking

access

tokens in the referer headers. Facebook’s engi-

neers could not provide us with more information about

what this entails or any direct responses they received,

but an SSOScan re-run one month later (early July 2014)

revealed that only four out of the 95 sites had ﬁxed their

problems (of the ten sites leaking access

tokens, only

two had been ﬁxed). Even for Facebook, it appears to be

difﬁcult to convince consumer-focused websites to take

security vulnerabilities seriously.

6.3 Deployment

Our experiences reporting vulnerabilities found by SSO-

Scan suggest that notifying vendors individually will

have little impact, which is consistent with experiences

reported by Wang et al. with on-line stores [26]. Hence,

we consider two alternate ways of deploying SSOScan

to improve the security of integrated applications.

App center integration. We believe SSOScan would

be most effective when used by an application distribu-

tion center (e.g. Apple store, Google Play) or identity

provider (e.g., Facebook) as part of the application vali-

dation process. The identity provider has a strong moti-

vation to protect users who use its service for SSO, and

could use SSOScan to identify sites that can compro-

mise those users. It could then deliver warning messages

USENIX Association 23rd USENIX Security Symposium 509

to visitors of vulnerable applications during the log in

through Facebook SSO process, or even go so far as to

shut down SSO logins for that application. We also be-

lieve our results can provide guidance to vendors devel-

oping SSO services. The results in Section 4.1 indicate

that sites are more likely to misuse credentials when us-

ing the Facebook JavaScript SDK. With Facebook’s help,

this problem could be mitigated by placing detailed in-

structions inside the SDK. The instructions could be pre-

sented as (non-executable) code in the SDK rather than

as comments, so that the developers cannot get by with-

out reading and removing them.

Checking-as-a-service. Without involving an central-

ized infrastructure, the best opportunity to deploy SSO-

Scan is as a vulnerability scanning service that devel-

opers can use to check their implementations before

their applications are launched (our prototype service at

http://www.ssoscan.org/ can be used for this now). For

a developer-directed test, it would be reasonable to ask

the developer to either guide the tool through the reg-

istration process or provide a special test account that

bypasses this step in cases where it cannot be fully au-

tomated. Even if we assume no aid from the developers,

they should at least be able to tolerate a longer testing

time than is feasible in doing a large-scale scan.

Availability

SSOScan is available at http://www.SSOScan.org/ as a

public web service. The source code is available (linked

from that site) under an open source license.

Acknowledgements

We thank Jonathan Burket, Longze Chen, Shuo Chen,

Steve Huffman, Jaeyeon Jung, Haina Li, Chris Slowe,

Ankur Taly, Rui Wang, Westley Weimer, Eugene

Zarakhovsky and anonymous reviewers for their valu-

able inputs and constructive comments. This work has

been supported by a Research Award from Google and

research grants from the National Science Foundation

and Air Force Ofﬁce of Scientiﬁc Research.

References

[1] N. Alshahwan and M. Harman. Automated Web

Application Testing Using Search Based Software

Engineering. In 26

IEEE/ACM International Con-

ference on Automated Software Engineering, 2011.

[2] G. Bai, J. Lei, G. Meng, S. S. Venkatraman, P. Sax-

ena, J. Suny, Y. Liuz, and J. S. Dong. AuthScan:

Automatic Extraction of Web Authentication Pro-

tocols from Implementations. In 20

Network and

Distributed System Security Symposium, 2013.

[3] T. Ball and S. K. Rajamani. The SLAM Project:

Debugging System Software via Static Analysis. In

ACM Symposium on Principles of Program-

ming Languages, 2002.

[4] C. Bansal, K. Bhargavan, and S. Maffeis. Discover-

ing Concrete Attacks on Website Authorization by

Formal Analysis. In 25

Computer Security Foun-

dations Symposium, 2012.

[5] M. Benedikt, J. Freire, and P. Godefroid. VeriWeb:

Automatically Testing Dynamic Web Sites. In 11

International World Wide Web Conference, 2002.

[6] BugBuster. BugBuster is a Software-as-a-Service

to Test Web Applications.

http://bugbuster.com/.

[7] C. Cadar and D. Engler. Execution Generated Test

Cases: How to Make Systems Code Crash Itself. In

International Conference on Model Checking

Software, 2005.

[8] G. Di Lucca, A. Fasolino, F. Faralli, and U. De Car-

lini. Testing Web applications. In Journal of Soft-

ware Maintenance, 2002.

[9] S. Elbaum, S. Karre, and G. Rothermel. Improving

Web Application Testing with User Session Data.

In 25

International Conference on Software Engi-

neering, 2003.

[10] Y.-W. Huang, S.-K. Huang, T.-P. Lin, and C.-

H. Tsai. Web Application Security Assessment

by Fault Injection and Behavior Monitoring. In

International Conference on World Wide Web,

2003.

[11] F. Nentwich, N. Jovanovic, E. Kirda, C. Kruegel,

and G. Vigna. Cross-Site Scripting Prevention with

Dynamic Data Tainting and Static Analysis. In 14

Network and Distributed System Security Sympo-

sium, 2007.

[12] G. Pellegrino and D. Balzarotti. Toward Black-Box

Detection of Logic Flaws in Web Applications. In

Network and Distributed System Security Sym-

posium, 2014.

[13] P. Pirolli, W.-T. Fu, R. Reeder, and S. K. Card. A

User-tracing Architecture for Modeling Interaction

with the World Wide Web. In First Working Con-

ference on Advanced Visual Interfaces, 2002.

510 23rd USENIX Security Symposium USENIX Association

[14] V. Rastogi, Y. Chen, and W. Enck. AppsPlay-

ground: Automatic Security Analysis of Smart-

phone Applications. In Third ACM Conference on

Data and Application Security and Privacy, 2013.

[15] Redspin Inc. Penetration Testing, Vulnerability As-

sessments and IT Security Audits.

https://www.

redspin.com/

[16] F. Ricca and P. Tonella. Analysis and Testing of

Web Applications. In 23

International Confer-

ence on Software Engineering, 2001.

[17] P. Saxena, D. Akhawe, S. Hanna, F. Mao, S. McCa-

mant, and D. Song. A Symbolic Execution Frame-

work for JavaScript. In 31

IEEE Symposium on

Security and Privacy, 2010.

[18] R. Sekar. An Efﬁcient Black-box Technique for De-

feating Web Application Attacks. In 16

Network

and Distributed System Security Symposium, 2009.

[19] Selenium development team. Selenium: Web ap-

plication testing system.

https://selenium.org/.

[20] J. Somorovsky, A. Mayer, J. Schwenk, M. Kamp-

mann, and M. Jensen. On Breaking SAML: Be

Whoever You Want to Be. In 21

USENIX Secu-

rity Symposium, 2012.

[21] S. Sprenkle, E. Gibson, S. Sampath, and L. Pol-

lock. Automated Replay and Failure Detection

for Web Applications. In 20

IEEE/ACM Inter-

national Conference on Automated Software Engi-

neering, 2005.

[22] S. Sprenkle, E. Hill, and L. Pollock. Learning Ef-

fective Oracle Comparator Combinations for Web

Applications. In International Conference on Qual-

ity Software, 2007.

[23] S.-T. Sun and K. Beznosov. The Devil is in the

(Implementation) Details: An Empirical Analysis

of OAuth SSO Systems. In 19

ACM Conference

on Computer and Communications Security, 2012.

[24] TestingBot. Selenium Testing in the Cloud - Run

Your Cross Browser Tests in Our Online Selenium

Grid.

http://testingbot.com/.

[25] R. Wang, S. Chen, and X. Wang. Signing Me

onto Your Accounts through Facebook and Google:

A Trafﬁc-Guided Security Study of Commercially

Deployed Single-Sign-On Web Services. In 33

IEEE Symposium on Security and Privacy, 2012.

[26] R. Wang, S. Chen, X. Wang, and S. Qadeer. How

to Shop for Free Online – Security Analysis of

Cashier-as-a-Service Based Web Stores. In 32

IEEE Symposium on Security and Privacy, 2011.

[27] R. Wang, Y. Zhou, S. Chen, S. Qadeer, D. Evans,

and Y. Gurevich. Explicating SDKs: Uncover-

ing Assumptions Underlying Secure Authentica-

tion and Authorization. In 22

USENIX Security

Symposium, 2013.

[28] Whitehat Security. Your Web Application Security

Company.

https://www.whitehatsec.com/.

[29] Q. Xie and A. M. Memon. Model-Based Testing

of Community-Driven Open-Source GUI Applica-

tions. In 22

IEEE International Conference on

Software Maintenance, 2006.

[30] L. Xing, Y. Chen, X. Wang, and S. Chen. In-

teGuard: Toward Automatic Protection of Third-

Party Web Service Integrations. In 20

Network

and Distributed System Security Symposium, 2013.

[31] Z. Yang, M. Yang, Y. Zhang, G. Gu, P. Ning, and

X. S. Wang. AppIntent: Analyzing Sensitive Data

Transmission in Android for Privacy Leakage De-

tection. In 20

ACM Conference on Computer and

Communications Security, 2013.

[32] C. Zheng, S. Zhu, S. Dai, G. Gu, X. Gong, X. Han,

and W. Zou. SmartDroid: An Automatic System

for Revealing UI-based Trigger Conditions in An-

droid Applications. In Second ACM Workshop on

Security and Privacy in Smartphones and Mobile

Devices, 2012.

[33] Y. Zhou and D. Evans. Technical Report: SSO-

Scan: Automated Testing of Web Applications

for Single Sign-On Vulnerabilities.

https://www.

ssoscan.org/SSOScan-TR.pdf